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Abstract of 
A Study of the Processes, Methods and Techniques used in 
Negotiated or “Best Value” Procurement of Construction Services 


An important part of the construction process 1s awarding the contract to perform the 
construction in a timely manner to a qualified contractor. The owner must choose a delivery 
method and a selection method. The four basic delivery methods are: Traditional Design-Bid- 
Build, Construction Management, Design-Build, and Design-Build-Operate-Transfer and the two 
most commonly used selection methods are Invitation to Bid and Request for Proposal (Source 
Selection). The method that will give the owner the most flexibility in selecting the most 
qualified contractor is the Request for Proposal or Source Selection Process. 

There are several basic steps in the Source Selection Process that ultimately lead to the 
selection of the "Best Value" Offeror. In the private industry the steps are left to the discretion of 
the individual owners. For Government Agencies there are applicable rules and regulations that 
apply and must be followed. In either case, the basic framework of the process is the same for 
both private and Government "owners". The primary difference is that private owners can enter 
into contractual relationships in any matter they choose as long as it is not fraudulent and 
therefore do not run the risk of the process being stopped by a protest. On the other hand, the 
Government runs the risk of the process being stopped by a protest and must follow all the 
prescribed rules, regulations, and procedures to mitigate this risk. 

The most important step in the Source Selection Process is developing Evaluation Factors 
and Subfactors because this will ultimately determine the ease and cost of proposal preparation 
for the offerors and the ease of evaluating proposals for the evaluation team. The second most 
important step is the consistent evaluation of the proposals received. No matter how complex the 
evaluation factors are if the evaluators evaluate the proposals consistently and adequately 
document their findings the probability of a successful protest will be significantly reduced. 

The Source Selection Process is the best available process to get the "best value" possible 
for any owner organization. It helps force contractors who want to continue to win contracts to 
care about their performance ratings on projects, helps reduce the adversarial relationships 
between the owner and the contractor, and in the end results in an overall higher quality project 


that is much easier to administer. 
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PART I 
Objective and General Process 


1.0 Objective of the Report 


The objective of this report is to describe and analyze the process, methods, and techniques 
owners use to select contractors in Negotiated or “Best Value” procurements for construction 
Services. 

The construction process from concept to project completion has many steps. Often times 
once the decision to go ahead with a project 1s made the time line for getting the project 
designed, awarded, and constructed is critical and can have significant economic impact if 
construction completion is not done by deadlines and/or within the most reasonable time 
possible. For example, I was involved in two school projects that had critical construction 
completion times and therefore, it was crucial they were awarded on time. The first project was 
the design build construction of a new school that had to be completed before the lease of the 
existing school expired, and the second project was a school renovation project where a 
significant portion of the work had to be done during the summer break. Another obvious 
example is for every day of delay in the completion of a hotel or apartment complex is 
significant in terms of lost revenue. 

One of the early phases of the construction process that can impact the time cycle and 
subsequent success (measured in quality and timeliness) 1s awarding the contract in a timely 
manner to a qualified contractor. This report will focus on the major steps in the selection by 
briefly describing each major step, comparing different agencies/organizations methods and 


procedure, reviewing case studies, and interviewing contractors and owner personnel. 





Additionally, the report will focus mostly on Government selection processes because it 1s 
much more prone to protests that can result in costly legal fees and considerably delay contract 
award and subsequent project completion. As per Robert Dorsey in CASE STUDIES IN 
BUILDING DESIGN & CONSTRUCTION, “private owners can select contractors and establish 
contractual relationships in almost any manner, short of fraud” (Dorsey, 1999; pg. 46). Public 
owners typically fall under strict regulations and procedures and when a selection decision does 
not appear fair, dissatisfied contractors can file a bid protest that will stop the entire acquisition 


process until the protest is resolved. 


2.0 General Pre-Award Process and Source Selection 


2.1 Selecting a Method to Choose a Contractor 


Once the decision 1s made to pursue the project the owner must decide on the delivery system 
to use. There are four basic delivery systems with variations within each major system: 
Traditional Design-Bid-Build, Construction Management, Design-Build, and Design-Build- 
Operate-Transfer. Also, at this point the owner should determine the selection method that they 
intend to use. There are two basic methods commonly used in the United States: Invitation to 
Bid (IFB) and Request for Proposal (RFP) 1.e. Source Selection. The invitation to bid 1s 
typically based on price and the lowest bidder receives the award provided that they can show 
that they have adequate financial resources to get the project done. In a request for proposal the 
owner develops guidelines and/or evaluation factors and asks the contractor to submit a proposal 
for completing the project based on the guidelines in the owner’s request for proposal. The 
owner then chooses a contractor based on the “best value” proposal received. 

Until recently invitation to bid has been the most popular method for procuring construction 


services. However, taking the lowest bid does not necessarily result in the lowest final cost and 





it does not typically give incentives for high quality. Therefore, Requests for Proposals have 
gained much wider use in the United States for example, the Naval Facilities Engineering 
Command (NAVFAC), awarded 35% of its contracts using RFP’s in fiscal year 1996, 40% in 
FY 1997, and is projected to reach 80% by the end of the FY 2000 (NFCTC, 1999). 

2.2 The General Source Selection Process 

Once an owner chooses to use Source Selection procedures then the process is widely 
dependent on the owner’s own internal policies and in the case of Federal and State “Owners” 
the laws and policies that apply to the process. Government “Owners” must follow stringent 
guidelines and almost always maintain “Full and Open Competition”, private owners have great 
latitude in determining how they will select contractors. However, the major steps in the process 
are similar for both Government and Private owners. 

The basic premise is that the owner develops a request for proposal (RFP) and then sends the 
RFP out to contractors asking them to submit proposals. The request for proposal can be as 
simple as a statement of work and some general guidelines on how to answer the proposal to 
detailed plans and specifications with very detailed instructions on how to answer the proposal. 
Often times there will be a request for two submissions, the first proposal submitted will be to 
pre-qualify contractors so the field of eligible contractors is narrowed down for the final and 
more detailed proposal submissions. The detail and complexity of the RFP will depend on the 
project and contract delivery method i.e. Design Build, Design-“Bid’’-Build, and/or Construction 
Management, and the rules and policies of the owner and/or government organization that 1s 
“buying” the project. 

The owner should develop the RFP with specific objectives and/or important points that they 


want the contractor to address that will help the owner discriminate between proposals. 





Typically, price is a factor, and then the request for proposal can specify other non-price related 
items ranging from site drawings and structural plans to Past Performance and/or a Plan to 
Manage the Project. The factors should relate to the important aspects of the project and/or show 
the contractor is capable, and how capable, he/she is to perform the project. 

The number of contractors the RFP is sent to depend on the owner. In the case of some 
organizations, such as the Walt Disney World Company, the requests are sent out to a selected 
group of contractors who have already pre-qualified with Disney through a pre-qualification 
process. For specific projects Disney sends out a request for proposal to 5 or 6 contractors on the 
pre-qualified list based on contractor skill and Disney’s perception of the workload of the 
contractor (Greenwood, Lee). In the case of the Federal Government the request has to be sent 
to the Commerce Business Daily to be publicly published and the public is invited to submit a 
proposal. The Florida Department of Transportation also uses an initial “pre-qualification 
process” but the process 1s limited to the contractors assets and does not address other factors 
such as Past Performance, personnel qualifications, etc. In fact they pre-qualify about 400 
contractors per year and when a contract is ready for “bid” letters of interest are sent to the list of 
contractors on the pre-qualified list (Moore, Juanita). 

Once Proposals are received, the owner and/or owners representatives meet, review the 


proposals, and ideally select the contractor that will provide the “best value” performance. 





3.0 Definitions 


Competitive Range. All proposals that the contracting officer determines to be the most 
highly rated based on the cost and noncost evaluation factors stated in the solicitation. 
Evaluation Factors and Subfactors. Descriptions of those aspects of a proposal that 
will be evaluated quantitatively or qualitatively to assess which proposal can best meet 
the Government’s requirements as described in the solicitation. 

Evaluation Standards (also called evaluation criteria). A baseline level of merit or 
acceptability used for measuring how well or whether an offeror’s response meets the 
solicitation’s requirements with respect to an evaluation factor or subfactor. 

Rating. The application of a scale of words, colors, or numbers, used in conjunction with 
narrative, to denote the degree to which the proposal has met the standard for a non-cost 
factor. 

Source Selection/Evaluation Plan. The document that explains how proposals are to be 
solicited and evaluated to make the selection decision. It is the Government’s plan for 
how it intends to acquire its needs. 

Source Selection Process. The process of soliciting and evaluating proposals for award 
in a competitively negotiated environment. 

Trade Offs. The process of choosing something that is a balance between price and 


technical merit. 





eis Val 
Analysis and Description of the Major 
Steps in a Source Selection 


4.0 Introduction 


The Source Selection Process consists of several basic steps that ultimately result in an award 
to the “Best Value” Offeror. In the private industry the steps are left to the discretion of the 
individual owners. For Government Procurements there are applicable rules and regulations that 
apply and must be followed throughout the process. In either case, the basic framework of the 
process 1s the same for both private and Government organizations. 

The following will primarily focus on Government source selections because the rules and 
regulations must be followed. If the process is not followed the end result could be a protest that 
could result in a complete work stoppage and much lost time. As stated earlier Private industry 


does not run this risk. 


5.0 The Source Selection Team 


Once a decision is made to use the Source Selection process a team must be established to 
help develop the Request for Proposal (RFP), evaluate the proposals, and recommend and select 
the best value contractor. It is very important to form the team early in the process. The Source 
Selection Team members should have input into the selection factors. The level of involvement 
of the Team members and the complexity of the team organization is usually determined based 
on the aspects of the project itself (AMC 715-3, 1998; NFCTC, 1998). 

The Team can be very structured and have many groups and subgroups to simply one or two 
people evaluating the proposals. AMC Pamphlet 715-3, CONTRACTING FOR BEST VALUE, 


A BEST PRACTICES GUIDE TO SOURCE SELECTION, has an example of different source 





selection teams. Figure | 1s an example of a complex Source Selection Team. Figure 2 is an 


example of a simple evaluation team. 


Source Selection 
Complex Source Selection Team 
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Contracting Officer FINAL DECISION 


Business Advisor to 
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Source Selection Advisory Council 
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Source Selection Evaluation Team 
Evaluates proposals against RFP Factors and Subfactors 
Assists SSA in comparative analysis, if required 





Advisors Advisors 
Government Non Government 
(if needed) (if needed) 


Technical Subgroup 
Evaluates Technical Merit 
and Proposal Risk 


Cost Subgroup Past Performance Subgroup 
Evaluates Cost Evaluates Performance Risk 





Figure | 





Source Selection 
Non-Complex Source Selection Team 







SOURCE SELECTION AUTHORITY 
(usually the Contracting Officer) 
Compares And Decides 













EVALUATOR 
Evaluates Cost 








EVALUATOR 
Evaluates Performance Risk 







EVALUATOR 
Evaluates technical Merit 
and Proposal Risk 






Figure 2 


The Naval Facilities Engineering Command has a typical Source Selection organization as 


shown in figure 3 (NFCTC, 1998). 









Source Selection Authority 
Makes Final Decision 








Source Selection Board 
Makes a recommendation to the 
Source Selection Authority 







Price/Cost Evaluation Board 
Evaluates price factors 


Technical Evaluation Board 
Evaluates all technical and 
performance factors 






Figure 3 
All of the above organizations are Department of Defense Organizations however committees 


and selection groups are formed for other organizations as well. 


1] 





All of the organizations I talked to included the “end-user” in the selection process. 
According to J.T. McAferty from the University of Florida’s Planning Department they short list 
their consultants, (A&E and Construction Managers), and after the short list the specific 
Department the facility is being constructed for within the University has a very significant role 
in making the final selection. Also, Lee Greenwood stated that they consult with the 
construction management branch and the “end-user” to choose the contractor. It is my 
experience in the Navy that many of the “end-users” are not involved in the detailed review of 
proposals but rather they receive briefs from the chairman of the Technical and/or Source 
Selection Board. However, this very much depends on the customer, type of project, and 
importance of the project. 

It is important to maintain the integrity of the team throughout the process. Once 
evaluations are started major changes to the team can be detrimental to the selection. Obvious 
changes can and should occur if a conflict of interest is discovered or other unforeseen 
circumstances arise among individual members. For example, on one selection I was involved 
with, after proposals were received and the technical evaluation board met and started reviewing 
proposals one of the technical board members noticed that his wife’s aunt held an executive 
position on one of the offeror’s board of directors. The technical board member notified the 
contracting officer and was appropriately removed from the technical evaluation board. In this 
case there were still several remaining members on the technical evaluation board and the board 
successfully completed the evaluations. 

However, care should be taken if major changes in the technical evaluation boards 


membership or procedures are changed as shown in the following case study. 
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5.1 Case Study United International Investigative Services vs. United States 


In United International Investigative Services vs. United States, 41 Fed. Cl. 312 (1998) the 
Court ruled in favor of the plaintiff when only 2 of the 6 members were called back to evaluate 
the proposals. The Contracting Officer felt that the Technical evaluations were flawed and 
directed the Technical Evaluation Board Chairman to revisit the proposals. The TEB Chairman 
directed two of the six TEB members to review the proposals and validate the scores. However, 
the record is not clear whether one or two of the TEB members actually did this. Whatever, the 
case the TEB member(s) who revisited the proposal made significant changes in the scoring. 
The court ruled in favor of the plaintiff because the court felt in part that 

The group discussion, as demonstrated above, was necessary 1n order to ventilate the 
evaluators’ opinions regarding each proposal. During these discussions, evaluators could 
have explained their individual reasons for ascribing a score to a particular proposal. 
Moreover, the TEB, as a group, would then have been able to arrive at a consensus 
regarding that particular aspect of an offeror’s proposal. 

Although minor irregularities or errors in the procurement process are not sufficient 
grounds to warrant judicial intrusion to overturn a procurement decision, Grumman Data 
Sys. Corp. v. Dalton, 88 F.3d 990, 1000 (Fed. Cir. 1996), the violation in this case was not 
minor. Rather, the violation described above deprived plaintiff of the opportunity to have 
its proposal considered fairly and honestly. Thus, the decision to have two evaluators 


circumvent the consensus and discussion requirements of the Technical Evaluation merits 
judicial intrusion. 


6.0 Source Selection Approach 
Once the decision is made to do a Source Selection the type of approach to use must be 
determined. There are several types of approaches and some examples of different approaches 
are described below: 
6.1 Pre-qualification - Submit Proposals 
In this approach the owner asks contractors to submit a pre-qualification proposal. The pre- 


qualification proposal usually only asks for general qualifications, past performance on similar 
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projects, and other company specific factors. Then the owner pre-qualifies usually three to five 
of the original offerors. The contractors who are pre-qualified will then get sent the full request 
for proposal including project specific evaluation criteria. 

This approach is usually more applicable to complex acquisitions, design/build type projects, 
and/or simpler acquisitions when there is an expectation that an excessive number of proposals 
will be received. 

The advantages to this are: 


a. It reduces the number of “complex” or detailed proposals that the owner has to 
evaluate and makes selection easier. 


b. It reduces the number of contractors who are submitting expensive proposals and 
eliminates contractors who would not have a very good chance at getting an 
award before they waste too many resources developing a proposal. 

The disadvantages are: 
a. It adds another step in the process and perhaps more time 1n the selection process. 
b. For simple projects it may not be appropriate. 
6.2 Submit Proposals 
The owner develops specific criteria and asks contractors to submit proposals. 


The advantages are: 


a. It eliminates a step and can be done 1n one evaluation phase and therefore in some 
cases can be done faster than the pre-qualification method. 


The disadvantages are: 


a. It can result in getting an excessive number of proposals and make it very 
difficult for evaluating the best value proposal. 


b. It can result in too many contractors investing resources in developing a proposal 
that they have no chance to a very slim chance of winning. 





6.3 Maintain List of Pre-qualified Companies ask for proposals 


Some private companies maintain a list of pre-qualified contractors. For example, the Walt 
Disney World Corporation maintains a database of contractors who have been pre-qualified to do 
work for them. The contractors must request to get on the list and then once they are approved to 
go on the list Disney will consider them for projects of the appropriate size and/or trade that the 
contractor is qualified to perform. For specific projects Disney invites 5 to 6 companies to 
submit proposals on the project from their pre-qualified list of contractors (Greenwood, Lee). 

For Federal projects a list of pre-qualified contractors such as Disney maintains would not be in 
compliance with 10 U.S.C. 2304 and 41 U.S.C. 253, which requires Federal Agencies to 
Maintain Full and Open Competition. Therefore, Federal Agencies would not be allowed to do 
this unless under very specific circumstances. 

The advantages to this are: 


a. Build a working relationship with contractors who know that if they perform well 
they will have a good chance for follow on work. 


The disadvantage 1s: 
a. May eliminate very qualified contractors who are not pre-qualified. 
6.4 Lowest Price Technically Acceptable 
This method is most likely a Government unique process. It 1s defined in the Federal 

Acquisition Regulation FAR Part 15.101-2. It is a process where contractors are asked to submit 
price and a technical proposal and the evaluator’s rate the technical proposal as either pass or 
fail. It is used when the best value is expected from the lowest price, technically acceptable 
proposal. 

The advantages 1s: 


a. It is much easier to evaluate proposals and therefore can result in a faster award. 





The disadvantages 1s: 

a. It does not allow for trade offs. A much more qualified contractor may have 
submitted a proposal for just slightly more money than a much less qualified 
contractor and not get the award 

For example, I was involved in a $500,000 warehouse/cold storage 
renovation project using this method and a contractor with an excellent 
proposal submitted a price that was less than 1% higher than a contractor 
who submitted a proposal that just met requirements. The contract award 
had to go to the “just met requirements” contractor even though it was clear 


the Government would have received better value with the slightly higher 
cost. 


6.5 Design Build Adjusted Score 
This method was described by Ken Leuderalbert, Florida Department of Transportation, 
Manager of Quality Initiatives. The method is used solely for selecting contractors for Design- 
Build projects in the Florida Department of Transportation. Basically, it 1s similar to the Pre- 
qualification - Submit Proposals, method described above but the technical evaluation panel 
assigns a percentage score to the technical proposals from Phase II. Then the submitted price is 
divided by the technical score and the lowest price wins. The table is a sample (not from a real 


project) of the “Adjusted Score”. 















Bid Price Technical 


Score 


Offeror 1 | 1,000,000 1,000,000/.8 1,250,000 


Offeror 2 | 1,200,000 276 1,200,000/.95 ie26en 7 


Offeror 3 | 900,000 900,000/.65 1,384,615 


Offeror 1 would be awarded this project. 


Calculation | Adjusted Bid 





The advantages are: 


a. It eliminates another level of proposal review where price and technical factors 
are considered together to determine best overall value. 


b. It helps alleviate appearances of favoritism. (The technical evaluation panel does 
not see the price during technical proposal reviews). 





The disadvatages are: 


a. It does not allow for trade offs between cost and technical merit and therefore 
the “Best Overall Value” contractor may not get the award. 


7.0 The Source Selection Plan 


Once the decision 1s made to use the Source Selection method a plan should be prepared to 
define the process and evaluation methods to be used. This plan actually starts with the 
formation of the team and choosing the delivery method. For small non-complex acquisitions 
the plan can be very simple but for larger more complex acquisitions it can be very complex. 

In Federal Acquisitions a source selection strategy or acquisition plan must be developed. 
Federal Acquisition Regulation Part 15.303(b) states in part that one of the requirements of the 
source selection authority is to, “Approve the source selection strategy or acquisition plan, if 
applicable, before solicitation release.” Many agencies have developed guidelines and 
requirements for what is commonly called the “Source Selection Plan”. 

The major purposes of most Source Selection Plans include; 

e Defining a specific approach for soliciting and evaluating proposals. 


e Describing the evaluation factors and subfactors, their relative importance, and the 
methodology used to evaluate proposals. 


e Providing essential guidance to the solicitation developers, especially for putting 
together the solicitation sections dealing with proposal preparation and evaluation. 


e Serving as a charter and guide for the source selection team on the roles of the 
members and the conduct of the entire source selection from proposal evaluation, 
through the cost/price/technical tradeoff, award decision, and debriefing (AMC 
Pamphlet 715-3, 1998). 


Appendix A is a blueprint for a Source Selection plan from AMC Pamphlet 715-3 and shows 


recommendations for the contents and basic layout of a source selection plan. 





Basically, as stated earlier the Source Selection Plan becomes a map for the entire 
procurement process from team selection to debriefing of unsuccessful offerors and should be 


the document used to develop the solicitation and evaluate proposals. 


7.0 Determining What and How to Evaluate Proposals 


8.1 Evaluation Factors and Subfactors 

Evaluation Factors and Subfactors are descriptions of those aspects of a proposal that will be 
evaluated quantitatively or qualitatively to assess which proposal can best meet the requirements 
as described 1n the solicitation. They are also commonly referred to as evaluation criteria. They 
are the basis of what the proposals will be evaluated on and how the selection will be made. 

For Federal Acquisitions all of the evaluation factors and subfactors must clearly be stated in 
the solicitation and the Source Selection Plan. The Federal Acquisition Regulation (FAR Part 
15.304) requires Cost and Past Performance to be evaluation factors (unless under very specific 
circumstances), and depending on dollar value the extent to which Small and/or Small 
Disadvantage Businesses are used. The Federal Acquisition Regulation Part 15.304 clearly 
states the requirements for evaluation factors and subfactors and is therefore included in part 


below. Key elements of this section are colored. 


15.304 Evaluation factors and significant subfactors. 


(a) The award decision is based on evaluation factors and significant subfactors that are 
tailored to the acquisition. 

(b) Evaluation factors and significant subfactors must-- 

(1) Represent the key areas of importance and emphasis to be considered in the source 
selection decision; and 

(2) Support meaningful comparison and discrimination between and among competing 
proposals. 

(c) The evaluation factors and significant subfactors that apply to an acquisition and their 
relative importance, are within the broad discretion of agency acquisition officials, subject 
to the following requirements: 





(1) Price or cost to the Government shall be evaluated in every source selection (10 U.S.C. 
2305(a)(3)(A)(11) and 41 U.S.C. 253a(c)(1)(B)) (also see Part 36 for architect-engineer 
contracts); 

(2) The quality of the product or service shall be addressed in every source selection 
through consideration of one or more non-cost evaluation factors such as past performance, 
compliance with solicitation requirements, technical excellence, management capability, 
personnel qualifications, and prior experience (10 U.S.C. 2305(a)(3)(A)(i) and 41 U.S.C. 
253a(c)(1)(A)); and 

(3)G) Except as set forth in paragraph (c)(3)(iv) of this section, past performance shall be 
evaluated in all source selections for negotiated competitive acquisitions expected to 
exceed $1,000,000. 

(11) Except as set forth in paragraph (c)(3)(iv) of this section, past performance shall be 
evaluated in all source selections for negotiated competitive acquisitions issued on or after 
January 1, 1999, for acquisitions expected to exceed $100,000. Agencies should develop 
nhase-in schedules that meet or exceed this schedule. 

(1) For solicitations involving bundling that offer a significant opportunity for 
subcontracting, the contracting officer must include a factor to evaluate past performance 
indicating the extent to which the offeror attained applicable goals for small business 
participation under contracts that required subcontracting plans (15 U.S.C. 
637(d)(4)(G)(1)). 

(iv) Past performance need not be evaluated if the contracting officer documents the reason 
past performance is not an appropriate evaluation factor for the acquisition. 

(4) The extent of participation of small disadvantaged business concerns 1n performance of 
the contract shall be evaluated in unrestricted acquisitions expected to exceed $500,000 
($1,000,000 for construction) subject to certain limitations (see 19.201 and 19.1202). 

(5) For solicitations involving bundling that offer a significant opportunity for 
subcontracting, the contracting officer must include proposed small business 
subcontracting participation 1n the subcontracting plan as an evaluation factor (15 U.S.C. 


637(d)(4)(G)(1)). 

Individual agencies also may require different evaluation factors to be considered and limit 
the number of factors that can be used. For instance, for certain acquisitions Sub-Contracting 
Plans and Sub-Contracting with Small and/or Small Disadvantaged Businesses is a required 
element in more stringent detail das prescribed in the Federal Acquisition Regulation. 
Interesting to note, The Disney World Company also considers Small and Small Disadvantaged 
Businesses when selecting contractors (Greenwood, Lee). 

The Naval Facilities Engineering Command’s Policy is to minimize the number of non 
cost/price evaluation factors with normally three to five. NAWFAC recommends that typical 


evaluation criteria include: 
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1. Past Performances 
2. Subcontracting with Small and Small Disadvataged Business 
3. Technical/Management Approach 
4. Key Personnel (NFCTC, 1998) 
These major factors seem to be very common with all of the agencies I interviewed including, 
Florida Department of Transportation., University of Florida, and the Army Corps of Engineers. 
AMC Pamplet 715-3 summarizes the purpose and importance of Technical evaluation factors. 

Technical evaluation factors address the proposal’s technical and performance 
efficiency and it is important to get the correct factors. The basic requirements for 
noncost evaluation factors are: 

e A reasonable expectation of variance among proposals in that area. 
e A variance that can be measured quantitatively or qualitatively. 
e The factor must be a true discriminator. 

An evaluation factor should be chosen only if your requirements warrant a 
comparative evaluation of that area. The simplest way to assess a potential evaluation 
factor is to ask: “Will superiority in this factor provide value to the Government and ts 
the Government willing to pay more for that superiority?” 

Selecting the right evaluation factors is one of the most important decisions you will 
make in designing your evaluation process. We are often faced with the triple problem 
of less time, less funds, and fewer available personnel to devote to source selections. If 
you don’t concentrate on what’s important in selecting the best value offeror you could 
end up with the evaluation team wasting a lot of time and effort looking at issues that 


don’t differentiate between offerors. This can also result in a weak evaluation that 
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doesn’t give the source selection authority the information needed to make a good 
selection. 

It is my experience that selecting the nght evaluation factors is not just one of the most 
important decisions but the most important decision in designing the evaluation process. 
Evaluation Factors must not only meet the criteria stated above but they also must be written in a 
clear and concise manner so the contractor clearly understands what the Government wants. | 
have been in many circumstances where unclear evaluation factors have led to difficulties and 
undo hardship on the evaluation committee and offerors. 


8.1.1 Selecting Evaluation Factors Do’s and Don’ts 





The importance of selecting appropriate evaluation factors can not be stressed enough. 
According to Vernon J. Edwards, a respected consultant in Government contracting: “I would 
suggest that the single most effective thing acquisition managers can to do streamline the best 
value source selection process 1s to improve their choices of evaluation factors for award.” 
(Savare, Matthew. www.monomouth.army.m1l/cecom/ac/hb/sourceselection!2. html) 

Almost all of the organizations that I interviewed said that when they started doing Source 
Selections they asked for too much detail. However, it seems from my interviews that most 
evervone, including the contractors were satisfied with the evaluation factors and/or criteria that 
is currently being used and did not see a need for major improvements. The following is a list 
of general lessons learned and or advice from the different organizations and/or my personal 
experience. 


e Only ask for the information that will discriminate between proposals and can be evaluated. 





This has been stated before however, it 1s worth repeating. Specifically, make sure that the 


evaluation factors will actually be of value in discriminating between proposals and/or actually 
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be an indicator of performance. Paul Nishaizaki, an architect from the NAVFAC Pacific 
Division, used the following example as an element under Past Performance that was not 
particularly valuable. He said that he has seen solicitations that ask for the data on completed 
contracts that included among other things: Original award amount, final contract amount, 
original contract completion date, and final completion date and that this information was not 
particularly useful. For most projects the original amount and completion date is different from 
the final contract amount and completion date and as an evaluator there was no way of knowing 
what the causes were for the differences without having personal knowledge and/or requiring the 
contractor to submit some kind of narrative description. 


e Insure Criteria is not inconsistent or spread out too much in the proposal. 





Mr. Richard Palmer, from “C” Construction Company, stated that sometimes it is difficult to 
really nail down all of the criteria because it 1s spread out all through the request for proposal. 
Mr. Larry Fisk, Naval Facilities Engineering Commands Southern Division, also stated that this 
caused difficulties 1n some of their proposals. In order to help alleviate his organization now 
develops a paragraph that identifies special project requirements and all of the evaluation factors 
refer back to that paragraph. This process 1s described in more detail in section 8.4. 

I was involved in a Design-Build Golf Course project that had criteria spread throughout the 
proposal. There were some minimum requirements that included in part: pump size and number, 
reservoir size and dimensions, number of sprinkler heads and types, number of holes that could 
be taken out of operation at a time, and the control system to be used. In addition, there were 
desirables that if included would be considered favorably. An example of one of the desirables 


was watering within some number of yards to the tee but closer was better. 
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The minimum requirements were spelled out in the solicitation but they were “spread out” 
throughout the solicitation. The technical evaluation board and it seemed offerors had some 
problem keeping all of the criteria organized and too make sure they met all of the minimum 
requirements while trying to meet the desirables. The technical evaluation board had to spend a 
lot of time going through the RFP and back to the proposals to make sure that all of the 
minimum requirements were met. The Technical Evaluation Board all agreed at the conclusion 
of the technical board that it would have been much better to have a succinct list of project 
requirements on one or two pages. 

e Do not be afraid to limit the amount of information. 

It is my opinion that this is most applicable to specific factors like; Past Performance, Safety, 
Quality Control, and other factors that are not as prone to innovations or creative solutions that 
would require significant explaining. I was involved in a RFP that was for the repair of a 
Typhoon damaged hangar. Most of the work was structural with some interior repairs. One of 
the sub factors under Past Performance was to demonstrate that the offeror had experience on 
similar type projects. Several of the offerors submitted extensive lists of projects and I 
remember specifically that one of the offerors submitted a list of 80 projlects ranging from 
runway lighting repairs to concrete wharf construction and repairs. The technical evaluation 
board had to wade through this information and determine what was applicable and what was 
not. After that solicitation our office started limiting the number of project to show experience to 
no more than 10 or 15 projects. Naval Facilities Engineering Commands Southern Division, 
limits the number of projects to 3 to 5 that relate specifically to the identification of special 
project requirements paragraph and also state in their proposals that they desire not to exceed a 


certain number of pages (Fisk, Larry). 
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In my opinion another example of a place to limit pages is the submission of cornorate Safety 
Plans or Quality Management Plans. These documents are usually large and can constitute a 
significant portion of the bulk of a proposal and do not necessarily provide clear discrimination 
between offerors. Most of the time corporate Safety Plans and Quality Control Plans contain 
basically the same information from one company to the next, however some companies do have 
unique aspects within these plans. But, for the most part they are the same. It is difficult fora 
technical evaluator to wade through all of the material in these plans and pick out unique aspects 


and/or valuable aspects that set contractors apart from one another. Our office changed our 


policy of asking for complete plans limiting the information to unique aspects and project 


accepted. It may not always be appropriate to limit information, like most factors it depends on 
the project type and complexity. 


e Make sure evaluation factors and submittal requirements are clear and specific. 





Mr. Nishaizaki, (NAVFAC Southdiv), stated that he has sometimes seen that after proposals 
are received that the government wasn’t clear on the types of submittals and/or clear enough on 
the evaluation factors by the content of the different proposals. Mr. Garicia of J.A. Jones 
Construction, stated that his office had submitted proposals in the past where they clearly 
misinterpreted the evaluation factors and that he would like to see presentations and/or 
interviews to insure a mutual understanding of requirements. 

It appears to me that most of the organizations that I talked to had already gone through the 
learning curve for developing and selecting evaluation factors and were doing much better at 


putting clear evaluation factors and submittal requirements in their solicitations. 
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8.2 Weighting the Factors and Subfactors 





Weighting of evaluation factors and subfactors is the “ranking” of importance of one factor 
over other factors. The Federal Acquisition Regulation requires that the relative importance of 
factors and subfactors be clearly stated (FAR PART 15.304(d)). This is not required in private 
industry. The Walt Disney World Company does not typically rank their evaluation factors; they 
review their proposals in the entirety and choose the best contractor with no formal ranking of 
evaluation factors (Greenwood, Lee). For Federal procurements the relative importance of 
evaluation factors and subfactors 1s usually established by priority statements, numerical 
weighting, or a combination of these. The FAR also clearly requires that the difference between 
cost/price and other non cost/price be clearly stated at a minimum as follows: 


(d) All factors and significant subfactors that will affect contract award and their relative 
importance shall be stated clearly in the solicitation (10 U.S.C. 2305(a)(2)(A)Q) and 41 
U.S.C. 253a(b)(1)(A)) (see 15.204-5(c)). The rating method need not be disclosed in the 
solicitation. The general approach for evaluating past performance information shall be 
described. 

(e) The solicitation shall also state, at a minimum, whether all evaluation factors other than 
cost or price, when combined, are-- 

(1) Significantly more important than cost or price; 

(2) Approximately equal to cost or price; or 

(3) Significantly less important than cost or price (10 U.S.C. 2305(a)(3)(A)(iu1) and 41 
U.S.C. 253a(c)(1)(C)). 


Naval Facilities Contracts Training Center lists examples of Priority Statements. Two of the 
seven examples listed are included below: 
Example 1. Priority Statement 


Proposals will be evaluated on technical/business qualifications and price. The 
technical/business qualifications factors will rank as the highest factors in this 
procurement, and are significantly more important than price, although price will also be 
important in the evaluation process. Within technical qualifications, there are three 
subfactors. These are corporate capabilities, key personnel and past performance data. 
Past Performance is of greater importance than the other two subfactors. Key personnel 
and corporate capability are of equal importance. 
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Example 2. Numerical Relationship 
The Government will make award to the responsible offeror(s) whose offer conforms to 
the solicitation and is most advantageous to the Government, cost or price and technical 
factors listed below considered. 
The evaluation factors listed below are in descending order of importance: 
(1) Past performance on similar projects 
(2) Technical and Management Approach 
(3) Experience on similar projects 
(4) Qualifications of key personnel 
(5) Price/cost (NFCTC, 1998) 
Whatever the priority statement is it must show the relationship of the importance of the 
individual factors. This becomes important when selecting the “best value” contractor. If the 
priority statement is weak or ambiguous it can cause different interpretations among the Source 
Selection Team and contractors and some confusion in choosing the “best value” contractor. 
8.3 Standards for Evaluation Factors 
Evaluators must be able to determine the relative merit of each proposal with respect to the 
evaluation factors. Evaluation standards provide guides to help evaluators measure how well a 
proposal addresses each factor and subfactor identified in the solicitation. Standards permit the 
evaluation of proposals against a uniform obiective baseline. This helps to minimize bias and 
promotes consistency in the evaluation by ensuring that the evaluators evaluate each proposal 
against the same baseline (AMC Pamphlet 715-3, 1998). 
Two examples of evaluation standards follow: 
This standard shall be met when the offeror provides evidence of a documented and 
functioning in-house training program for the handling, transport and disposal of 
hazardous waste in accordance with EPA and State guidelines and regulations. (NFCTC, 
1998) 
STANDARD FOR PAST PERFORMANCE has been met when the offeror provides at least 
10 of any combination of the following: Performance evaluations for past contracts, letters 


of recognition, letters of appreciation, recognition awards, and any other client 
documentation that recognizes acceptable performance on past projects. 





When our office first started doing Source Selections on a regular basis we did not include 
evaluation standards. I was involved with about 6 of these Source Selections and after we 
incorporated evaluation standards it did in fact eliminate some of the confusion and personal 


interpretation problems that we were having in earlier solicitations. 


8.4 Using Identification of Snecial Proiect Reauirements Paragraphs 





‘The Southern Engineering Field Division, Naval Facilities Engineering Command 
(SOUTHDIV) use identification of special project requirements as one of the basis to evaluate 
offeror’s proposals. The solicitation has a paragraph that clearly states the special project 
requirements and then the evaluation factors are developed so they relate back to the special 
requirements (Fisk, Larry). 

For example, they issued a solicitation for a 2 Phase Design Build Wharf Utilities Project in 
Mayport Florida. The special project requirements paragraph said that experience with electrical 


distribution systems among other things was required. The evaluation factors were all related 


back to the special proiect requirements paragraph; past performance, technical qualifications, 


offerors filed a GAO Protest, Durocher Dock & Dredge/Black & Veatch, A Joint Venture, 
B280853, November 24, 1998, in short stating that they were improperly evaluated and should 
have been ranked higher and included in the second phase. However, the GAO denied their 
protest stating in part: 


The TEB assigned Durocher’s proposal a rating of marginal under this 
factor. Under the two subfactors evaluated in this area--design team 

and construction team--offerors were instructed to submit key 

personnel staffing that demonstrated experience in the elements 

included in the Specialized Project Requirements provision of the RFP. 
Offerors were also required to provide qualifications and staffing 
capability to support the project. 
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Under the design team subfactor, the TEB found that only one individual Durocher 
proposed as “key personnel” had participated in any of the projects Durocher described 
in its proposal (and even that individual had participated in only one project described 
in the proposal), and identified this as a weakness in the proposal. The TEB further 
found that the proposed key personnel demonstrated no waterfront electrical 
distribution, steam, or fuel system experience and considered this a weakness in the 
proposal. Under the construction team subfactor, the TEB found that the proposed key 
personnel demonstrated no waterfront electrical distribution experience, and that the 
waterfront experiences listed for key structural personnel were minimal in scope. The 
protester does not rebut any of the TEB’s findings in this regard. 


In its proposal, Durocher identified the firm of [DELETED] as a mechanical team 
member and [DELETED] as the electrical installation team member. The TEB noted, 
however, that the protester did not include any personnel data for [DELETED], and 
found that the proposal demonstrated no specific diesel fuel or steam work experience 
by construction project managers, superintendents, or quality assurance personnel. 


This GAO case summary 1s included in its entirety in Appendix B. It should also be noted the 
case 1s also an excellent example of a TEB supporting their decision with a solid narrative. 

The below is a partial example of a special projects requirement provision of a solicitation 
currently in Phase II of the solicitation. 


This project involves the site-adapting design and construction of a standard US Navy T- 
10 aircraft jet engine test cell facility to support intermediate aircraft engine maintenance 
for F/A —18 squadrons at Marine Corps Air Station, Beaufort, SC. These test cell facilities 
more closely resemble a working piece of machinery than a typical brick and mortar type 
project. They are designed and constructed to operate without failure when subjected to the 
stresses induced by continuous operation of high-powered aircraft jet engines within the 
engine run room. The test cell construction involves fabrication and installation of complex 
facility equipment and instrumentation. The facility equipment and instrumentation to be 
provided closely integrates with NAVAIR furnished and installed equipment, 
instrumentation test systems and a variety of aircraft jet engines. The construction of a T- 
10 jet engine test cell involves modular prefabricated steel and sheet metal type fabrication. 
The major components of the facility are jet engine run room, primary and secondary air 
inlets, movable primary air inlet turning vanes, exhaust augmenter and exhaust stack, all 
constructed with acoustically treated panels and materials. The augmenter will be the US 
Navy T-10 standard test cell air cooled round type used in other similar facilities around 
the world. The test facility and especially the augmenter are subjected to continuous 
vibration, acoustical loading and pressure variations. In addition the augmenter 1s subjected 
to a wide range of variations in internal temperatures from ambient up to 2,800+F and 
airflow velocities in excess of 2,500 feet per second. The quality of the fabrication of the 
facility and especially the augmenter system is extremely critical to ultimate success of this 
test facility project. The manufacture and installation of the prefabricated aircraft engine 
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test facility is desired to be performed by a manufacturer with an adequate history of 
proven successful experience in the fabrication of complex close tolerance prefabricated 
steel/metal type structures who is regularly engaged in this type of work, has the 
equipment, tools, technical personnel and experienced shop personnel for the fabrication of 
close tolerance prefabricated structures, and demonstrates adequate shop space for 
fabrication and shop assembly/factory demonstration of the test facility indoors. Other 
complex systems include air start system, fuel and oil preservation systems, engine test 
trailer restraint system, proof load test system, fire protection, industrial waste, power and 
lighting. Supporting facilities will involve replacement of the existing air start compressor, 
installation of 20,000-gallon underground fuel tank, fuel oil separator, civil site work and 
utilities. The estimated price is between $5,000,000 and $10,000,000. Proposals will be 
evaluated using Two Phase Design-Build Request for Proposal procedures that will in 
Phase IT result in an award of a firm-fixed price design-build contract to the responsible 
offeror whose proposal, conforming to the solicitation, is the “Best Value” to the 
Government, price and technical factors considered. Phase I is the pre-qualification phase 
wherein offerors will be evaluated on: FACTOR A — Past Performance for Design and 
Construction; FACTOR B — Small Business Subcontracting Effort (applies to small and 
large business); and FACTOR C - Technical Qualifications in Design and Construction 
and Factor D - Management Approach. The most qualified offerors (not to exceed 5) will 
be selected to proceed into Phase II . The Phase II Request for Proposals will be issued 
only to the pre-qualified offerors. Phase II will include the Price Proposal and an additional 
technical factor Technical Solutions, which will require submission of a limited design 
solution for the project site work and other sub-factors that define the quality of 
construction. (www.esol.navfac.navy.mil, N62467-99-R-1055) 


According to Mr. Larry Fisk, this paragraph was designed to help streamline the Source 
Selection Process and so far has worked well. It appears that this approach helps offerors insure 
that their proposals are focused on important factors and helps evaluators focus on what is 
important in an offeror’s proposal. 


8.5 Establishing a Rating Method 





The rating method is the “score” that each evaluation factor will receive. Each rating must 
have a definition. There are 3 common rating methods when developing a proposal and AMC 


Pamphlet 715-3 is a summary of the methods and the descriptors: 


Zo 


_ Typical Ratings and Descriptors 





Each rating must must have a definitio on. 


TECHNICAL MERIT ratings reflect the government’ s confidence in each offeror’s ability, as demonstrated in | 
its proposal, to perform the requirements stated in the RFP. Choose one method (e.g., numerical, adjectival, or _ 
color) to evaluate technical merit. 

NUMERICAL | ADJECTIVAL — 1 GOLOR DEFINITION 


90-100 | | Excellent | Blue | Proposal demonstrates excellent understanding of 
: | requirements and approach that significantly exceeds 
performance or capability standards. Has exceptional 
| strengths that will significantly benefit the 
Government. 





80-89 |Good — 













| Proposal demonstrates good understanding of 
] requirements and approach that exceeds performance | 
or capability standards. Has one or more strengths that | 
| S| Sa | __} will benefit the Government. 
70-79 ) Satisfactory Yellow | Proposal demonstrates acceptable understanding of 


| requirements and approach that meets performance or 
| capability standards. Acceptable solution. Few or no 

| strengths. _ 

} Proposal demonstrates shallow understanding of : 
| requirements and approach that only marginally meets | 
| performance or capability standards necessary for : 
| minimal but acceptable contract performance. 
| Fails to meet performance or capability standards. 
Requirements can only be met with major changes to 
jthe proposal. 


| Marginal 









<60 | Unsatisfactory 


Rating systems which use adjectives or colors are usually the most successful because they 
allow maximum flexibility in making the tradeoffs among the evaluation factors (AMC Pamphlet 
715-3). However, many organizations use a numerical system. Ms. Claudia Hunley, Army 
Corps of Engineers Jacksonville, stated that the numerical system is easier to explain and easier 
for members of Technical Evaluation Boards to understand and for this reason they use a 
numerical rating system. 

A narrative definition must accompany each rating in the system so that evaluators have a 


common understanding of how to apply the rating. For example, a rating of excellent (or blue or 
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90-100) could be defined as meaning an outstanding approach to specified performance with a 
high probability of satisfying the requirement. What is key in using a rating system in proposal 
evaluations, 1s not the method or combination of methods used, but rather the consistency with 
which the selected method is applied to all competing proposals and the adequacy of the 
narrative used to support the rating (AMC Pamphlet 715-3, 1998). 

Each major rating is described below: 


Adiectival 
Adjectives (such as excellent, good, satisfactory, marginal, and unsatisfactory) are used to 


indicate the degree to which the offeror’s proposal has met the standard for each factor 
evaluated. Adjectival systems may be employed independently or in connection with other rating 
Systems. Holmes & Edwards 


Color Coding 
This system uses colors to indicate the degree to which the offeror’s proposal has met the 


standard for each factor evaluated. For instance, the colors blue, green, yellow, amber, and red 
may indicate excellent, good, satisfactory, marginal, or unsatisfactory degrees of merit, 
respectively. 


Numerical 
This system assigns point scores (such as 0-10 or 0-100) to rate proposals. This rating system 


generally allows for more rating levels and thus may appear to give more precise distinctions of 
merit. However, numerical systems can have drawbacks as their apparent precision may obscure 
the strengths, weaknesses, and risks that support the numbers. Therefore, some organizations do 
not permit the use of numerical rating systems. 


Narrative 
Narrative is used in conjunction with a rating system to indicate a proposal’s strengths, 


weaknesses, and risks. Adiectival, color, and numerical ratings must be supported with narrative 
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statements. Narrative statements can describe the proposals’ relative strengths, weaknesses, and 
risks to the source selection authority in a way that adjectives, colors, and numbers alone cannot. 
A narrative 1s required when evaluation standards are being applied, when a comparison of 
proposals is being made, and when a cost/technical tradeoff 1s conducted. The narrative provides 
a reasonable and rational basis for the selection decision (AMC Pamphlet 715-3). 

Different organizations may or may not require the use of specific rating methods and the 
three rating methods above are not the only rating methods that can be used. Examples of 
different requirements are: NAVFAC Policy is to use an adjectival system, the Air Force uses a 
color system, and the Florida DOT uses a numeric system that ends in an overall percentage 
score for their technical proposals. 

8.6 Summary 

Once a solicitation is issued the most important thing is to follow what 1s said in the 
solicitation in terms of evaluation factors, evaluation factor weight, standards, and the rating 
method and support the selection decision with adequate documentation. Developing the factors 
will have the most significant impact in terms of ease of evaluation, time required to evaluate 
proposals, and reduction in the probability of a protest. Making it easier to evaluate and identify 
strengths and weaknesses by choosing the right evaluation factors, weights, standards, and rating 
methods will ultimately make it easier, faster, and safer (from protests) to make an award to the 


Best Value contractor. 
9.0 The Solicitation 


The solicitation provides all of the information the offeror needs to understand what you are 
buying, how you are buying it, and how you will select who to buy it from. This information 


Date 


includes: the work requirements; the terms and conditions; evaluation factors and significant 


32 


a 
: — = 


— nt 
7 LD 




















a ie al 
is i, acme: ie mmm 
—— —_ —- eee es > —_— 


<< @ 


—  — a 


subfactors; the relative importance of the factors and subfactors; instructions to offerors, 
including whether award might be made without discussions; and other exhibits and attachments. 
When read as a whole, the solicitation should convey to the offerors a clear understanding of 
what you are buying and the areas where technical and cost tradeoffs can be made in their 
proposals to best satisfy the Government requirements. 

The solicitation 1s what the contractors will see and what they will use to develop their 
proposals. When developing the solicitation consistency across the board is important. There 
should not be conflicts between the description of what we are buying, instructions on how to 


prepare a proposal, and guidance on important factors/subfactors , and the ground rules for the 


evaluation (AMC Pamphlet 715-3, 1998). 
10.0 Evaluating Proposals 


Evaluating proposals is the process in which each proposal 1s evaluated in detail against the 
evaluation factors and subfactors and the requirements set forth in the solicitation, and assigning 
a rating with a supportive narrative. It is not an analysis of proposals against each other. For 
Government projects it is very important that evaluations be fair, thorough, and impartial. It 1s 
also important to stress detailed analysis. The Technical Evaluation Board must be detailed in 
their review and reach a consensus in their ratings of the proposals. 

Additionally, as stated in section 8.5, the technical evaluation board must provide narrative 
Statements to establish a written record. Numerical scores and other rating techniques are not 
conclusive data to make the source selection decision. Only evaluations and ratings 
Piscranriated by documented specific strengths, weaknesses, and risks can be credible and 
justifiable. General terms such as “weak”, “poor”, or “excellent” must be supported with 


specific reasons as to why the proposal is “weak”, “poor”, or “excellent” in relation to the 
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standard for the specific factor and/or subfactor being evaluated. A technical evaluation that 
does not substantiate proposal strengths, weaknesses, and risks will not be able to stand un 1n a 
protest (AMC 715-3, 1998). 

10.1 GAO Case Studies 

I have chosen two, of many GAO cases, that demonstrate the importance of the evaluating the 
proposals according to what is prescribed in the solicitation. It 1s worth noting that there are a 
large number of cases that demonstrate this and that this is probably the single biggest factor for 
protests that are sustained against the government. For more extensive review of GAO cases the 
GAO’s web site contains a complete list of all the cases they have reviewed since 1996. The site 
is www. ga0.gov. Each of the case summaries described below are included in Appendix B. 


10.1.1 Unfair Evaluation of Pronosals 





GAO Case Tidewater Homes Realty, Inc., B-274689, Dec. 26, 1996 
In this case the GAO found that there was an unequal evaluation of the proposals in 


solicitation by the Department of Housing and Urban Development (HUD) and sustained the 


factors the protestors received a score of 6 while the awardee’s proposal received a score of 24 
points. The technical evaluation panel asserted that the protestor’s proposal lacked detail and 
clarity in the factor, however, when the GAO reviewed the proposals they found that the 
information in each proposal was essentially the same. For another technical factor the agency 
downgraded the protestors proposal because they said it was vague in certain areas. However 
GAO’s review of the factor showed just the opposite. 

It 1s difficult to determine what led to the protest and missed interpretation of the 


proposals. It could have been a lack of detailed review, not following the evaluation factors, 
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and/or bias toward one contractor on the part of the Technical Evaluation Panel. However, it 
clearly shows the need to do a detailed analysis of the proposals so key factors or points are not 
missed. 


10.1.2 Not Following Established Criteria, inadequate documentation 





GAO Case Lloyd H. Kessler, Inc., B-284693, May 24, 2000 

In this case the agency did not perform the evaluation as stated. In fact the agency’s actual 
evaluation differed from that stated in the RFP in that the relative weights for experience and 
past performance were reversed and experience was assigned 70 points and past performance 
was assigned 30 points. Additionally, the agency rated very highly for experience in 
bioengineered slope protection experience when in the RFP it was never mentioned to be 
important. Additionally, in the notes of this case ( #5) the GAO alludes to the importance of 
documenting the selection decision and the GAO states, “even beyond the agency’s failure to 
disclose the actual evaluation plan in the RFP, the documentation supporting the evaluation 1s 
minimal, such that it would be difficult to determine the reasonableness of the evaluation, even if 
the RFP were not defective.” 

10.2 Oral Presentations 

Oral presentations are another method of presenting proposals. It must be determined early in 
the process 1f oral presentations are going to be used and clear and precise instructions must be 
put into the solicitation for conducting the oral presentations. Also, the evaluation 
criteria/factors should be explicitly stated and clear so offerors know what information is 
expected. (AMC Pamphlet 715-3, 1998: NFCTC, 1998) 

For Federal Contracting it 1s very important to make sure the content of the oral presentations 


are documented. There are many sustained cases where during oral presentations the 
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government did not adequately document the presentations and therefore there is not an adequate 
record of performance. A recent example 1s /&/J Maintenance, Inc., B-284708.2; B-284708. 3, 
June 5, 2000. The contract was for housing maintenance and service and the GAO sustained the 
protest stating in part, 


In this procurement, the proposals addressing the quality factor consisted of oral 
presentations, briefing slides, and resumes. No written technical proposals were allowed. 
[°] While the RFP stated that the agency could videotape and/or record the oral 
presentations, the agency did not do so. RFP amend. 9, at L-6. In this case, the offerors’ 
slides and the evaluators’ notes are the only record of what was included in the orai 
presentations. We recognize the SSO’s discretion to decide the method and level of detail 
of the record of an orai presentation and that the use of offerors’ slides and government 
notes are two of several possible methods of documenting an oral presentation under FAR 
§ 15.102(e). Here, however, the slides and notes do not present sufficient information to 
determine if the evaluations of J&J’s and D&Z’s proposais were conducted in a reasonable 
and equal manner. The slides provide only an outline--in most cases, the slides include 
general headings oniy--and do not describe what was included in the approximately 2-hour 
oral presentations and question-and-answer sessions. [*] The evaluators’ notes are not 
summaries of the presentation but, for the most part, are selective comments which are 
sketchy and provide almost no elaboration, or even a description, of what was contained in 
the portion of the oral presentation being commented upon by the evaluator. 


In addition, the consensus evaluation ratings consist only of numerical scores for each 
offer in each evaluation element, without any accompanying narratives to show how the 
panel reconciled the divergent opinions of the individual evaluators on any particular facet 
of an oral presentation or how the panel achieved a consensus rating for any element of the 
evaluation. There is no evidence in the consensus ratings sheets (or elsewhere in the 
record) that the evaluators discussed the strengths and weaknesses of the proposals or used 
any other method to reconcile their differences to arrive at the consensus scores. Agency 
Report, Tabs I, J, Consensus Ratings for Quality Proposals (D&Z and J&J, respectively). 


We understand that oral presentations are an effective way to streamline the source 
selection process and to enhance the agency’s understanding of an offeror’s approach, and 
we do not believe that the FAR requirement for documentation described above necessarily 
limits this flexibility. FAR §§ 15.102(e) and 15.308 do not require a particular method of 
establishing a record of what was said by offerors during oral presentations or place an 
excessive burden on contracting agencies in recording oral presentations. Whatever 
method is chosen, FAR §§ 15.102(e), 15-305(a) and 15-308 do establish an obligation to 
provide a reasonably adequate record of such presentations and the evaluation thereof. 
Such a record permits our Office to perform a meaningful review of the agency’s selection 
decision. Here, the oral presentations constituted the offerors’ entire technical proposals 
(the only written portions of proposals related to past performance). The record of the oral 
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resentations and the evaluation 1s so sketchy. that we have no means to determine. based 


upon the record before us, the reasonableness of the agency’s selection. 





The other key elements in conducting oral presentations are: 
e Limit the time for each presentation. 


e Some of the proposal should be in writing, ie hard data such as performance history or 
anything that will be included in the contract documents. 


e If exchanges are going to be allowed during the presentation then it is important that 
they are fair and meaningful for each offeror (AMC Pamphlet 715-3, 1998). 


The use of oral presentations depends on the agencies/organizations that I talked to. For 
instance, Ms. Claudia Hunley, ACOE, said they use them when they feel that it is appropriate, 
and said as a contracting officer she liked using oral presentations. Mr. Phenix Palmer of FDOT 
and Mr. Larry Fisk (SUOUTHDIV) said they prefer not to use oral presentations. The University 
of Florida extensively uses oral presentations when using the Source Selection Process. 

10.3 Summary 

The technical evaluation of the offeror proposals 1s the foundation of the source selection 
process and the pending contract award. It 1s also the source of the biggest areas for contractors 
to protest. The development of good evaluation factors and subfactors will significantly effect 
the complexity of the evaluation process and ease of evaluation. The key points in evaluating 
proposals are: 

e Evaluations must be fair, thorough, and impartial. 
e Document specific strengths, weaknesses, and risks of the proposed approach. 


e It must be done in accordance with the requirements stated in the RFP and the Source 
Selection Plan. 


e The evaluation must provide enough detail for determining merits between proposals 
and enough detail for the contracting officer to make a best value determination and or 
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establish a competitive range of offerors for further consideration. (AMC Pamphlet 715- 
Bees; NFCTC 1993) 


Even if the evaluation factors are not very good and/or hard to evaluate if the above points are 


followed the risk of a successful protest are low. 


11.0 Exchanges with Offerors After Receipt of Proposals 

In Federal Contracting there are basically three types of exchanges as per FAR 15.306; 
clarifications and award without discussions (Clarifications), communications with offerors 
before establishing the competitive range (Communications), and exchanges with offerors after 
establishment of the competitive range (Discussions). 

Clarifications are informal and limited to correction of minor deficiencies, clerical errors, as 
well as to give the offeror the opportunity to resolve adverse past performance issues. 
Communications are exchanges with an offeror whose past performance is the only factor 
preventing them from getting into the competitive range. Discussions are where the offerors 
will be allowed to revise their proposals. It is the actual “negotiating” proposals and prices can 
be changed and bargaining can take place. 

Discussions must be meaningful and fair. The following must be insured when conducting 
discussions: 

e Identify all deficiencies in the proposal 
e Specify all deficiencies to the offeror 
e Provide a reasonable time for revision 


e Make a complete record of the discussions 


e Hold discussions with all other offerors in the competitive range (AMC 
Pamphlet 715-3, 1998; NFCTC, 1998) 
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12.0 Selection Decision 


For Federal Acquisitions the final award decision belongs to the Source Selection Authority. 
According to AMC Pamphlet 715-3, the source selection authority must ensure the selection 
decision; 

e Is based on a comparative analysis of the proposals 


e Is consistent with stated evaluation factors and subfactors:; and 


When tradeoffs are permitted, consider whether or not perceived benefits are 
worth any price premiums. 


Make the decision on a rational basis and set it forth in an independent, stand- 
alone defensible document. 


The minimum information that is needed to document the award should include: 
e A description of the acquisition 
e The names of offerors 
e A summary of the strengths and weaknesses of each proposal and offeror 


e The reasons why the selected contractor provides the greatest probability of 
satisfying the Government’s requirements. (AMC Pamphlet 715-3, 1998, 
NFCTC, 1998) 


The below table is the AMC Pamphlet 715-3’s Blueprint for the source selection decision 


document. 


Documenting the Selection Decision and Awarding the Contract 
|____Source Selection Decision Document for (specify product/service & REP i) 
11. Decision Statement, | 
| Example: As Source Selection Authority for this acquisition, I have determined that the XYZ 
| product/service proposed by Offeror C provides the best overall value to satisfy Army needs. 
| This selection was made based upon the factors and subfactors established in the solicitation and | 
my integrated assessment and comparison of the strengths, weaknesses, and risks of the 
| proposals submitted in response to the solicitation. This memorandum documents the basis for 
| my decision. 


eed = = a — — 


39 












—— 
[— 


- —— = << “! 47 om. 7 


wt Ize’) @ == = 


. — = 


SS 


i 
» 





> <-> 


|2 2. Brief description of the product/service called for in the solicitation. | 


| 3. Brief description of the Basis for Award including the major factors against which | 
| proposals were measured and their relative order of importance. 


4 A list of offerors in the competitive range. 


: 5. | Rationale for business judgments and ‘tradeoffs. ‘Include the following: | | 
je Succinctly compare each proposal to each of the others, focusing on key proposal differences 

| (strengths, weaknesses, and risks) that surfaced in the evaluation and their impact on the 
| acquisition objectives. _ ae a 

| * Clearly explain specific tradeoffs that led to the decision. —~ | | oe 
¢ Clearly explain the specific benefits of technically superior offeror(s) and why they at are or are 
pots leniicant enough to warrant any additional cost. 













| Example: In summary, based on my integrated assessment of all proposals in accordance with 
| the specified evaluation factors and subfactors, it is my decision that Offeror C’s proposal offers 
the best overall value. 





) Signature 
(ROUSE Ci I 


13.0 Notification and Debriefing of Unsuccessful Offerors 


The FAR 15.503 requires that within three days after the date of contract award, the 
contracting shall provide written notification to each offeror whose proposal was in the 
competitive range but was not selected for award. This notice must include; 


(1) The number of offerors solicited: 

(11) The number of proposals received; 

(111) The name and address of each offeror receiving an award; 

(iv) The items, quantities, and any stated unit prices of each award. If the number of 
items or other factors makes listing any stated unit prices impracticable at that time, 
only the total contract price need be furnished in the notice. However, the items, 
quantities, and any stated unit prices of each award shall be made publicly available, 
upon request; and 

(v) In general terms, the reason(s) the offeror’s proposal was not accepted, unless the 
price information in paragraph (b)(1)(iv) of this section readily reveals the reason. In no 
event shall an offeror’s cost breakdown, profit, overhead rates, trade secrets, 
manufacturing processes and techniques, or other confidential business information be 
disclosed to any other offeror. 
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It is my experience that this 1s not a difficult step, however, if paragraph (v) is done with 
enough detail, it may help contractors understand why they did not get the award and eliminate 
the need for post award debriefings. For some solicitations we edited the technical evaluation 
report and sent the contractor the comments from the Technical evaluation report that pertained 
to their own proposal. The contractors felt that this information was very useful and helpful. 

The Federal Acquisition Regulation also requires post-award debriefings. These debriefings 
are usually chaired by the Contracting Officer and supported by members of the Selection Team. 
The debriefing can be oral, in writing, or by any other method that is acceptable to the 
Contracting Officer. At a minimum the Federal Acquisition regulation part 15.506 requires that 
the debriefing an offeror who responds on time (within three days of written notification of 
contract award) shall be debriefed and furnished the basis for the selection decision and contract 
award. It also states that, 


(d) At a minimum, the debriefing information shall include-- 

(1) The Government’s evaluation of the significant weaknesses or deficiencies in the 
offeror’s proposal, if applicable: 

(2) The overall evaluated cost or price (including unit prices) and technical rating, if 
applicable, of the successful offeror and the debriefed offeror, and past performance 
information on the debriefed offeror: 

(3) The overall ranking of all offerors, when any ranking was developed by the 
agency during the source selection; 

(4) A summary of the rationale for award; 

(5) For acquisitions of commercial items, the make and model of the item to be 
delivered by the successful offeror; and 

(6) Reasonable responses to relevant questions about whether source selection 
procedures contained in the solicitation, applicable regulations, and other applicable 
authorities were followed. 

(ec) The debriefing shall not include point-by-point comparisons of the debriefed 
offeror’s proposal with those of other offerors. Moreover, the debriefing shall not 
reveal any information prohibited from disclosure by 24.202 or exempt from release 
under the Freedom of Information Act (5 U.S.C. 552) including-- 

(1) Trade secrets; 

(2) Privileged or confidential manufacturing processes and techniques; 

(3) Commercial and financial information that is privileged or confidential, including 
cost breakdowns, profit, indirect cost rates, and similar information; and 
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(4) The names of individuals providing reference information about an offeror’s past 
performance. 
(f) An official summary of the debriefing shall be included in the contract. 

It is my experience that during debriefings the contractors want to know how they can do 
better and sometimes what the other contractors did to score higher than them. It is my opinion 
that it is important for the Government to divulge as much information as possible to help a 
contractor improve their proposal, but it is imperative that the Government representatives stay 
within the bounds of the offeror’s own proposal. For instance, one of the evaluation factors we 
often used was something like how will you manage quality on the project. Some contractors 
took an approach of describing their quality control organization and their companies 
commitment to quality and how they manage quality overall and by doing this they would show 
that they knew how to manage quality on any project. This is fine and was usually graded 
“Acceptable” unless they really could show a stand out quality organization. Another contractor 
may have gone into their overall organization and quality control procedures briefly but then 
really focussed on how they would insure quality on specific aspects of the project and be rated 
“Highly Acceptable”. During a debriefing the offeror who was rated “acceptable” would want to 


know how he could improve his rating. The government representative may be able to tell him 


to be more project specific next time but can’t really say we want vou to do it like offeror B. 


commitment to safety” and the offeror stated they have a safety incentive program but failed to 
describe it and received an "Acceptable" rating. The Government evaluator can tell them they 
could improve their proposal by describing their incentive program in more detail and that Just 
stating that you have an incentive plan does not merit a high grade unless you can show its 


effectiveness and/or value to the employees. 
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Mr. Richard Palmer of “C” Construction Company, Inc., said that he would very much like it 
if the Government could provide more information on debriefings and maybe at least show the 
“orades” or ratings other contractors received. He said that it was his opinion that there is some 
perception in the industry that sometimes the Government Agency has a contractor in mind and 
that contractor is given some partial treatment. He felt that if all of the “ratings” were shown It 
might alleviate that perception somewhat and help contractors gauge how they are doing 
compared to other contractors. 


Ms. Claudia Hunley, ACOE, stated that debnefings were very valuable for both the 


Government and the contractors and that the Government was able to learn how to make it easier 


to prepare and evaluate proposals. 
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Part Il 
Summary 


14.0 Contractors View 


The overall views of the contractors that I interviewed were consistent. Mast of the 


contractors felt that the process was fair but some felt that the personalities and/or agency could 


and sometimes did bias the system. The major points from the contractors were: 


High Cost - Preparing proposals, especially Design-Build, is expensive. The range of 





rough cost I received was from $25K for smaller jobs to $100K for the more complex 
solicitations (Garcia, Richard; Pepper, David). Mr. David Pepper of Harry Pepper & 
Associates, Inc. said that he felt the high cost made it difficult for smaller contractors to 
compete. 

It is worth noting that the Florida DOT provides stipends for the contractors that are in 
the 2" Phase of the 2 Phase Source Selection Process on Design Build projects to help 
offset the cost of preparation of the proposals. Mr. David Pepper and Mr. Jerry Stanley of 
Hughet Construction said that not getting a stipend definitely discourages potential offerors 


from competing on certain projects. 


Better coordination between owner and end user/facility tenant (Specifically Desien- 





Build) - Both Mr. Richard Garcia and Mr. David Pepper stated that the end user/tenant 
does not really have a clear picture of what the end product is. Once the contract is 
awarded and the final designs and/or construction starts the tenant finds that they thought 
they were getting something else, usually something better, than what was actually in the 


contract. Additionally, Mr. David Pepper said that he much preferred the Design-Bid- 
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Build process partly because the A&E does not have the face to face time with the end 
users when developing the plans and specifications. 

It is interesting to note, as stated in section 5.0, that the end users are included in all of 
the agencies/organizations selection teams. It is my experience in the Navy that this 
problem is inherent 1n the planning, design and/or development of the scope of work, and 
that the perceived lack of coordination 1s caused by many factors outside the scope of this 
paper. 

Evaluation Factors/Cniteria is reasonable - All of the contractors felt that the evaluation 
factors/criteria have been developed and are reasonable and did not feel that there were any 
major problems with them. However, Mr. Knoll of J. A. Jones Management Services did 
say that he felt if there were too many factors/criteria and subfactors that the important 
discriminators in a solicitation were lost. I specifically expected contractors to say that 
they felt the some factors were overly burdensome and didn’t understand what the 
agency/organization needed. However this was not the case. It 1s apparent that the 
agencies that I interviewed have fairly quickly learned what factors/criteria work well and 


don’t work well. 


Criteria and/or Proposal strengths not enforced - Mr. Richard Palmer of “C” Construction 





Co. Inc. said that the design criteria needed to be more prescriptive and/or better defined. 
He felt that sometimes the government would require something and than later relaxes the 
criteria. For example, the government may specify some minimum sound rating criteria 
and than accept a proposal that really did not meet the minimum sound rating cniteria. Mr. 
Knoll stated that he felt some contractors strategy was to tell the government what they 


want to hear to win but then not actually follow through and perform the work they 
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promised in their proposals. He also said that sometimes the proposals would be evaluated 
favorably for things that once the contract was awarded and execution started, the post 


award side of the contract administration did not feel was important. 


@ Other General comments - 
e Don’t require color renderings they are very expensive 
e Don’t change the format of required information from one solicitation to the next. 

Mr. Palmer, said that one agency would ask for Past performance in one prescribed way 
or on a certain form for one solicitation and then ask for the same information in a different 
format on the next solicitation. He felt it would be much better if the government would 
be specific about what they wanted and leave the format up to the individual contractors. 

e Don’t require the same information to be submitted twice. 
For example, requiring information on Past performance in the pre-qualification stage and 
then requiring the same information again from the contractors on the short list. 
e Debriefings are very useful and helpful. 
e Increased use of Oral Presentations would be beneficial to help insure that contractors 


understand requirements. 


15.0 Conclusions and Recommendations 


The Source Selection Process continues to gain popularity and is seeing increased use. All of 
the owner representatives that I talked to felt that it helped decrease adversarial relationships and 
improve the quality of construction/service. In fact, Mr. Larry Fisk (SOUTHDIV) said their 
office has almost completely abandoned the IFB processes because the acquisition/pre-award 


time is not that much different from Source Selection and the dividends in performance are 
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worth the extra time. He said that an average IFB takes 61 days to award as compared to 75 days 
for simple source selections. 

The process is still continuing to develop and improve. Different Agencies/organizations 
have different policies and procedures and have different levels of experience with the process. 
From my interviews experience seems to bring about the biggest process improvements and 
almost evervone I talked to said that at first they asked for too much detail and/or asked for 
information that did not discriminate between offerors. 

Arguably the most important step in the pracess 1s the selection and development of 
evaluation factors. If the evaluation factors are clear enough and as simple as possible without, 
of course, hampering offeror’s innovation and creativity the selection of the “Best Value” 
contractor will be much easier. If the evaluation factors are unclear and/or complex there is a 
preater risk of the evaluation getting bogged down and taking tao much time, confusion amongst 
offerors and evaluators, and contractor protests. 

The second most important step is the evaluation of the proposals itself. Even if the 
evaluation factors are complex and/or unclear if the evaluation team rates the proposals 
according to the evaluation factors and rate them consistently the conclusions and 
recommendations will stand in the face of a GAO protest or Federal Court proceedings. For this 
reason the Technical Evaluation Team should have some training so they understand the rules 
and guidelines they must follow and their importance in the source selection process. 

I believe that the Source Selection process is the best available way to get the best value 
possible for Government construction projects. It allows for the use of non-price related factors 
that are not available with the IFB process. It helps to force contractors who want to continue to 


win contracts to care about their performance ratings on projects, helps reduce the adversarial 
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relationship between the Government and contractors and tn the end results in an overall higher 


quality project that is much easier to administer. 
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APPENDIX A 


Guidelines for a Source Selection Plan 
from 
AMC Pamphlet 715-3 
Contracting for Best Value 
A Best Practices Guide to Source Selections 


Source Selection/Evaluation Plan 


Section I. Summary Description 





A. Item/Service to be Acquired 
B. Proposed Contract Type (Choose one) 


» FFP 

__ FP with Incentives 
__ Cost 

__ Cost with Incentives 
__ Hybrid (Explain) 


C. Source Selection Process (Choose one) 


__ Tradeoff 
__ Low Price Technically Acceptable 
__ Hybrid (explain) 


D. Special Evaluation Techniques (explain, if applicable) 
e.g. oral presentations, multi-step techniques 


Section II. Source Selection Team 


A. Members. List the evaluation team members by name and functional area (including 
advisors, if applicable). 


B. Team Member Certifications: Verify/attach by reference certificates from each team 
member protecting unauthorized release of source selection or proprietary information. 





Section III. Evaluation Factors and Subfactors 


A. Proposal Evaluation Information. State all factors, subfactors and their relative order of 
importance exactly as they will appear in the solicitation. If you elect to include desirable 
objectives or features you would be willing to pay extra for, include an explanation of how they 
will be evaluated and whether or not credit will be given for exceeding such desirables. 


B. Proposal Submission Information. State the instructions to offerors for preparing and 
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proposals exactly as they will appear in the solicitation. 
Section IV. Description of Evaluation Procedures 


| A. Rating System. Describe the evaluation technique (ratings and their definitions) that will be 
used to rate the proposals. 


B. Evaluation Standards. Describe the standards or target levels you will use to measure how 
well a proposal addresses each evaluation factor and subfactor. 


C. Procedures for Recording the Evaluation 
e.g. Worksheet(s) for recording evaluator ratings with supporting narrative 


D. Schedule. Identify the schedule for key source selection activities. 
Section V. Rationale for the Evaluation Scheme 


A. Factors and Subfactors. Explain how the selected factors and subfactors reflect the 
circumstances of your particular acquisition and the results of market research and other 
presolicitation exchanges with industry. 


B. Weights. Explain how the relative importance of the factors and subfactors reflect the 
acquisition’s primary objective. Explain why the most heavily weighted factors are the “drivers” 
| of the source selection. 
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APPENDIX B 
GAO CASE DECISIONS 


BNUMBER: B~280853 

DATE: November 24, 1998 

TITLE: Durocher Dock & Dredge/Black & Veatch, A Joint Venture, B- 
280853, November 24, 1998 
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DOCUMENT FOR PUBLIC RELEASE 
The decision issued on the date below was subject to a GAO Protective 
Order. This redacted version has been approved for public release. 


Matter of:Durocher Dock & Dredge/Black & Veatch, A Joint Venture 
Pale: B~280853 
Date:November 24, 1998 


mecer J. Ippolito, Esq., Hillyer & Irwin, for the protester. 

Mark Christopher, Esq., Lis B. Young, Esq., and George N. Brezna, 
Esq., Naval Facilities Engineering Command, for the agency. 

Aido A. Benejam, Esq., and Christine S. Melody, Esq., Office of the 
General Counsel, GAO, participated in the preparation of the decision. 


DIGEST 


1. Allegation that agency improperly evaluated protester's proposal 
is denied where the record shows that the agency evaluated the 
proposal in accordance with the evaluation factors announced in the 
solicitation and record reasonably supports protester's overall lower 
technical rating. 


2. Disparities in evaluation ratings among technical evaluators do 
not establish that the evaluation process was flawed or not rationally 
based in view of the potential for disparate subjective judgments of 
different evaluators on the relative strengths and weaknesses of 
technical proposals. 


3. Allegations by protester whose phase one proposal was reasonably 
ranked eighth, that agency improperly evaluated the first- and 
fourth-ranked proposals is dismissed where, even if protester's 
allegations were sustained, protester's proposal would not be eligible 
to proceed to phase two of the procurement; protester is not an 
interested party to pursue this aspect of the protest. 


DECISION 


Durocher Dock & Dredge/Black & Veatch, A Joint Venture protests the 
rejection of its proposal under request for proposals (RFP) No. 
N62467-98-R-0968, issued by Lhe Department of Lhe Navy Lo improve 
Wharf D, at the Naval Station in Mayport, Florida. The protester 
argues that the evaluation of its proposal was inconsistent with the 





SOLLGlGaclon. 
We deny the protest. 


The Navy issued the RFP on May 20, 1998, as phase one of a two-phase 
procurement under the procedures set out at Federal Acquisition 
Pegm@latvon clloparte30.5. KEP “sec. O02Z0Z;,7 amend. No. 0001, part 1.1.2, 
The RFP is for the design and construction of a major waterfront 
improvement project at the Naval Station, Mayport, Florida. The 
solicitation advised offerors that the agency would evaluate proposals 
in two phases. Under phase one, at issue here, the RFP listed the 
following evaluation factors in descending order of importance: past 
performance, technical qualifications, management approach, and small 
business subcontracting effort. Id. Based upon the results of that 
evaluation, the Navy would then select a maximum of five of the "most 
highly-qualified offerors" to submit phase two proposals, which are to 
be evaluated baséd on technical considerations and price. Award is to 
be made to the offeror whose proposal is deemed to represent the best 
value to the government, considering price and technical factors. Id. 


Twelve firms, including the protester, responded to the RFP by the 
time set on June 19, for receipt of phase one proposals. A technical 
evaluation board (TEB) evaluated proposals by assigning adjectival 
ratings (exceptional, acceptabie, marginal, or unacceptable) under 
each evaluation factor, and an overall proposal rating. Based on the 
overall ratings, the TEB then ranked proposals from the most 
highly-rated (exceptional) to the lowest-rated (unacceptable) as 
follows: 


[DELETED] Exceptional 
[DELETED] Exceptional 
Offeror A Acceptable 
{DELETED] Acceptable 
Offerom. Acceptable 
Of FerousC Marginal 

Offeror D Marginal 


Durocher Dock & Dredge/Black & VeatchMarginal 


CLTrerores Marginal 
Offeror F Marginal 
Offeror G Marginal 
Offeror H Unacceptable 


Agency Report at 2. 
Based on the results of the phase one evaluation, the TEB found that 
the five highest-ranked proposals had demonstrated superior past 


performance and technical qualifications over the remaining seven 
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firms, and recommended to the source selection board (SSB) that the 
firms that submitted those proposals advance to phase two of the 
procurement. The SSB accepted that recommendation and by letters 
dated July 16, the agency informed all offerors whether they were 
selected to proceed to phase two. Following a debriefing by the Navy 
conducted on July 29, the protester filed an agency-level protest, 
which the Navy denied. This protest to our Office followed. 


PROTESTER'S CONTENTIONS 


The protester argues that the evaluation of its proposal was flawed 
and inconsistent with the RFP in several respects. For instance, 
Durocher arques that the TER deviated from the evaluation scheme 
announced in the solicitation by emphasizing certain aspects of the 
work described in the solicitation (primarily electrical work), but 
that the Navy failed to notify offerors that this portion of the work 
would be considered more important in the evaluation. The protester 
aiso Itainmtains that individual evaluators were not consistent in 
applying the RFP's evaluation criteria. Durocher further argues that 
the agency improperly evaluated the proposals submitted by two firms 
selected to proceed to phase two of the procurement. 


DISCUSSION 


Our Office will not engage in an independent evaluation cf proposals 

nor make an independent determination of their relative merits. 

MeecOUeoy Ss, (nC, S-239123, AUG. te oO lOO 2 Cr) (para«ws Li4 ar 9 
Rather, we review the agency's evaluation only to ensure that it was 
reasonable and consistent with applicable statutes and regulations as 
well as with the terms of the solicitation. Sensis Corp., B-265790.2, 
Jan. 17, 1996, 96-1 CPD para. 77 at 6. A protester's mere disagreement 
with the agency's conclusions does not render the evaluation 
mmieeasOnable. ESCO; Inc., B—=Z2Z25565, Apr. 29, 1987, 8/-1 CPD para. 450 at 
7. Based on our review of the record, including the TEB's narrative 

in support of its evaluation, we conclude that the factor and overali 
ratings assigned the protester's proposal are reasonably supported. 

Below we discuss a representative sample of the TEB's findings with 
respect to the protester's proposal in support of our conclusion. 


Past Performance 


The TEB assigned Durocher's proposal a rating under this factor of 
"acceptable minus" (A-), [1] indicating the consensus of the TEB that 
the proposal was acceptable, but contained some weaknesses in this 
area. Under both subfactors in this area--design team and 
construction team--offerors were required to submit past performance 
narratives for up to three projects that demonstrated experience in 
performing work similar to that described in part I.1.3 of section 
GO202 of the RFP, the Specialized Project Requirements.[2] Offerors 
were also instructed to describe up to three partnering/teaming 
arrangements with design and construction teams. 


The TEB identified several strengths in the protester's proposal under 
both subfactors in this area. For instance, the TEB found that all 
three project narratives Durocher submitted demonstrated work relevant 
to the Wharf D improvement project in that they all were major 
waterfront projects, and their scope included most or all of the work 
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elements required to accomplish this project. The TEB identified 
several weaknesses, however. For example, the TEB found that the 
projects Durocher submitted for the design and construction teams did 
not contain medium voltage electrical distribution work (as described 
fiewart I.1.3 of section 00202 of the RFP). The TEB also found that 
Durocher's proposed design and construction teams demonstrated no past 
performance for steam systems, and had minimal experience with fueling 
systems. One evaluator specifically noted that the proposed design 
team's projects did not indicate any steam system design experience, 
and that it was unclear from the proposal whether the designers had 
performed any work related to fuel systems on any of the three 
projects Durocher described in its proposal. Technical Evaluation 
Worksheets. The TEB further found that none of the Key personnel in 
Durocher's proposed design team had worked on any of the partnering 
arrangements the firm described in its proposal. In our view, the 
TEB's consensus rating of A- under this factor reasonably reflected 
the evaluators’ concerns that Durocner's proposal had not demonstrated 
experience in performing work similar to that described in the 
Specialized Project Requirements section of the RFP. 


Durocher argues that by downgrading its proposal for failing to show 
that the projects it submitted contained medium voltage electrical 
distribution work, the TEB gave this weakness greater significance 
than announced in the RFP. In this connection, Durocher argues that 
the TEB improperly emphasized the "voltage power distribution system" 
aspect of the work over other aspects, and that the Navy failed to 
inform offerors that this work element was significantly more 
important than any other aspect of construction or design. 
Protester's Comments, Oct. 2, 1998, at 2. 


The protester's argument that the TEB emphasized one element 
(electrical distribution system) over other elements of the work is 
not supported hy the record. As already stated, the record shows that 
the TEB considered Durocher's proposal acceptable under this factor, 
but came to a consensus regarding the several weaknesses noted above. 
Since the TEB concluded that the projects Durocher submitted did not 
involve medium voltage electrical distribution work--one of the 
elements contained in part I.1.3 of section 00202 of the RFP--the TEB 
reasonably downgraded Durocher's proposal slightly under this factor 
to a rating of A-. The fact that the TEB found that this element was 
missing did nol mean Lhal it was given more imporlance Lhan oLlhers, 
and based on our review of the evaluation, we have no basis to object 
io the TEB's rating. 


The protester points out that some individual evaluators rated 
Durocher's proposal as exceptional in this area, while other 
evaluators downgraded its proposal for the medium voltage power 
distribution weakness. According to Durocher, this variation in 
individual ratings demonstrates that the members of the TEB were not 
uniform in applying the RFP's evaluation criteria. 


It is not unusual, however, for individual evaluators to have 
disparate judgments regarding the relative strengths and weaknesses of 
peeecniarcal proposal. “oyscen Servs a,eunc., S-255647,. Sept. 21, 1989, 
69-2 CPD para. 258 at 5. Disparities in evaluator ratings do not 
establish that the evaluation process was flawed or otherwise not 
reasonable. U.S. Defense Sys., Inc., B-245006.2, Dec. 13, 1991, 91-2 





CPD para. 541 at 4. Contrary to the protester's argument, the record 
shows that only one evaluator rated Durocher's proposal exceptional 
while the other evaluators rated Durocher's proposal either marginal 

or acceptable in this area, and that their narrative comments merely 
reflect the individual evaluators' subjective judgments regarding the 
relative strengths and weaknesses of the proposal in this area. In 

any case, notwithstanding the different individual ratings, the TEB 
reached a consensus in this area. 


Technical Qualifications 


The TEB assigned Durocher's proposal a rating of marginal under this 
factor. Under the two subfactors evaluated in this area--design team 
and construction team--offerors were instructed to submit key 
personnel staffing that demonstrated experience in the elements 
included in the Specialized Project Requirements provision of the RFP. 


Offerors were also required to provide quaiifications and staffing 
capability to support the project. 


Under the design team subfactor, the TEB found that only one 
individual Durocher proposed as "key personnel" had participated in 
any of the projects Durocher described in its proposal (and even that 
individual had participated in only one project described in the 
proposal), and identified this as a weakness in the proposal. The TEB 
further found that the proposed key personmnei demonstrated no 
waterfront electrical distribution, steam, or fuel system experience 
and considered this a weakness in the proposal. Under the 
construction team subfactor, the TEB found that the proposed key 
personnel demonstrated no waterfront electrical distribution 
experience, and that the waterfront experiences listed for key 
structural personnel were minimal in scope. The protester does not 
rebul any of the TEB's [findings in this regard. 


In its proposal, Durocher identified the firm of [DELETED] as a 
mechanical team member and [DELETED] as the electrical installation 
team member. The TEB noted, however, that the protester did not 
include any personnel data for [DELETED], and found that the proposal 
demonstrated no specific diesel fuel or steam work experience by 
conslLruclLion projecl managers, Superinlendenls, or quaiily assurance 
personnel. 


In its comments on the agency report, Durocher argues that both firms 
it identified in its proposal as (mechanical and electrical) team 
meinbers have previously worked on Navy projects, including Mayport. 
According to the protester, therefore, the Navy should have been 
familiar with these firms based on their performance on those 
projects. 


Durocher's reliance on the previous projects performed by its proposed 
team members without regard to the specific information required by 
the RFP is misplaced. A procuring agency's technical evaluation is 
dependent upon the information furnished in the offeror's proposal. 
Computerized Project Management Plus, B-247063, Apr. 28, 1992, 92-1 
CPD para. 401 at 3. An agency is not required to overlook a flawed 
proposal on the basis of the offeror's prior performance; on the 
contrary, all offerors are expected to demonstrate their capability in 
their proposals. Pedus Bldg. Servs., Inc., B-257271.3 et al., Mar. 8, 
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1995, 95-1 CPD para. 135 at 4. Conseguently, the agency reasonably relied 
on Durocher's proposal in determining the identified weaknesses in its 
qualifications and staffing capability to support the project. [3] 
Since Durocher has not shown that the evaluated weaknesses are 
unfounded based on the information contained in its proposal, we have 
no basis to object to the evaluation. [4] 

Based on our review of the evaluation record, including the 
protester's proposal and the TEB's individual worksheets, we think 
that the agency reasonably concluded that Durocher's proposal omitted 
material information regarding its past performance and failed to 
demonstrate its technical qualifications as required by the RFP. [5] 
Under these circumstances, we have no basis to object to the TEB's 
overall rating of marginal of the protester's proposal. 


The protester also argues that the agency improperly evaluated the 
proposals submitted by two firms selected to proceed to phase two of 
the procurement. In this regard, Durocher maintains that [DELETED] 
"has been in business for only five months and has yet to successfully 
complete a project." Protester's July 29, 1998 letter to the Navy at 
1-2. Durocher reiterates this argument in its comments on the agency 
report, maintaining that the record "reveals [DELETED] was formed in 
October of 1997." Protester's Comments, Oct. 2, 1998, at 4. 


Peaacition, Durocher asserts that {DELETED] "has lost virtually all 
of its management personnel that .. . were responsible for the 
construction and management of all projects listed in their 
performance record." Letter from Protester to the Navy at 2 (July 29, 
1998). According to Durocher, "it is obvious that a number of 
[DELETED] key employees left [DELETED] and became employees of the 
mewly formed [DELETED] contractor corporation in 1997." #Protester's 
Comments, Oct. 2, 1998, at 5S. The protester thus questions the higher 
ratings assigned the proposals submitted by [DELETED] and [DELETED}. 


Under the bid protest provisions of the Competition in Contracting Act 
@r 1994, Sl Wes. C.Aue Sec. “Soo l-SooG (West Supp. 197s, only an 
"interested party" may protest a federal procurement. That is, a 
protester must be an actual or prospective offeror whose direct 
economic interest would be affected by the award of a contract or the 
failure to award a contract. Bid Protest Regulations, 4 C.F.R. sec. 
21.0(a) (1998). Determining whether a party is interested involves 
consideration of a variety of factors, including the nature of issues 
raised, the benefit of relief sought by the protester, and the party's 
status in relation to the procurement. Black Hills Refuse Serv., 
Be2204/70, Peb. iho, 19688, 68-1 CPD para, 15) at 2-3. A protester is not an 
interested party where it would not be in line for contract award were 
Bee LOLeSL UO Heusustalted. HCa Composites, Inc., B=-235e49.2, Jan. 
by 1oo0, 20-1 Cro “para. 7 at 1. 


Based on their overall ratings, [DELETED] proposal (exceptional) was 
ranked first, [DELETED] proposal (acceptable) was ranked fourth, while 
Durocher's proposal (marginal) was ranked eighth overall. Since we 
conclude that the evaluation of Durocher's proposal was reasonable, 
even assuming that [DELETED] and [DELETED] proposals were downgraded 
so as to be eliminated from phase one, Durocher's proposal's relative 
ranking would rise only from eighth to sixth place. Since the RFP 
stated that a maximum of five of the "most highly qualified offerors" 





would advance to phase two, RFP sec. 00202, amendment No. 0001, part 
T.1.2., Durocher would not be eligible to proceed to phase two of the 
competition (instead, the higher-rated proposals submitted by offerors 
C and D would now be among the most highly-rated proposals). 


Where, as here, there are intervening offerors with a greater interest 
in the procurement than the protester, we generally consider the 
protester's interest to bé too remote to quaiify the protester as an 
interested party. Four Seas and Seven Winds Travel, Inc., B-244916, 
Nov. 15, 1991, 91-2 CPD para. 463 at 4. Since nothing in Durocher's 
protest would alter the ratings of the intervening offerors (Offerors 
C and D), Durocher would not be eligible to proceed to phase two of 
the competition even if its protest allegations were sustained; 
accordingly, Durocher is not an interested party to challenge the 


evaluation of the [DELETED] and [DELETED] proposals. 
The protest is denied. 


Comptroller General 
of the United States 


1. The TEB added a plus, a minus, or no designation to the factor 
ratings based on the strengths and weaknesses of each factor. To 
assist in determining proposal rankings, the TEB also added a plus, a 
minus, or no designation to the overall proposal ratings. 


2. Part 1.1.3 of section 00202 of the RFP describes the project, 
including the main design and construction elements such as dredging, 
structural, mechanical, electrical, and civil utilities. 


3. We recognize that in some cases, where an offeror's proposal refers 
Lo informalion cegarding tks pasl performance and Lhe information ts 
personally known to the evaluators, that information must be 
considered in the evaluation. See, e.g., International Bus. Sys., 
mere o—-2/5554, March 3, 1997, 97-1 CPD para: 114 at 5 ("Some information 
is simply too close at hand to require offerors to shoulder the 
inequities that spring from an agency's failure to obtain, and 
consider, the information"). That is not the case here. On the 
contrary, Durocher'’s proposal merely named its two proposed electrical 
and mechanical team members, and generally stated that their selection 
was based on their "knowledge of the Naval Station Mayport, and 
experience and familiarity with government contracting." Durocher's 
Proposal at 13. The proposai did not identify any Navy contracts at 
the Mayport facility with which the firms were involved and did not 
describe the services those firms rendered in connection with those 
contracts; nor does the proposal identify the specific activity for 
which the work was performed. 


4. Regarding the evaluation of [DELETED] proposal, one of the firms 
which was retained in the competition, Durocher argues that the TEB 
improperly considered as a strength information regarding a proposed 
electrical and mechanical subcontractor that was apparently not 
submitted in that firm's proposal. In this regard, the TEB found as 
one Of nine strenglhs under Lhe Pasl Performance faclor Lhal, 
"fajJlthough not submitted, .. . [the] (elec. & mech. sub.) has 
received several excellent commendations from SOUTHDIV for relevant 
work." Technical Evaluation Board Report at 13. Even assuming that 
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the TEB should not have considered that information, given the other 
strengths noted and the superior factor ratings assigned the proposal 
in other areas, there is no reason to conclude that the "exceptional" 
rating of [DELETED] proposal, or the overall proposal rating of 
"exceptional minus," would change. 


5. The TEB assigned Durocher's proposal a rating of "marginal plus" 
under the third evaluation factor, management approach. Offerors were 
required to submit an organizational chart for design and construction 
teams demonstrating contractual arrangements and lines of authority 
between key personnel. RFP, amend. No. 0001, part I.1.5, factor C. 
The TEB found that while the organizational charts Durocher included 
im its proposal listed all design and construction Key personnel, the 
charts and matrices presented were unclear. Also, the TEB concluded 
that the narrative included with the charts did not demonstrate an 
adequate understanding of the design-build process. Based on our 
review of the charts, the accompanying explanation, and the TEB's 
narrative in support of its evaluation, and since Durocher has not 
rebutted any of the evaluators' findings, we think the TEB's rating in 
this area is reasonable. 





BNUMBER: B-274689 
DATE: December 26, 1996 
TITLE: Tidewater Homes Realty, Inc. 
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Matter of:Tidewater Homes Realty, Inc. 
Bile: B-274689 
Date:December 26, 1996 


Freida V. Rapp and Kenneth S. Rapp for the protester. 

Virginia Kelly Stevens, Esq., and Jane D. Atkinson, Department of 
Housing and Urban Development, for the agency. 

Charies W. Morrow, Esq., and James A. Spangenberg, Esq., Office of the 
General Counsel, GAO, participated in the preparation of the decision. 


DIGEST 


1. Agency's best value award selection is not reasonably supported 
where the record reflects that the selection of the awardee was based 
on an unequal evaluation of the protester's and awardee's proposals. 


2. Agency improperly waived definitive responsibility criterion which 
required the awardee possess a Virginia Real Estate Broker License at 
the time of award, where the agency determined the awardee to be 
responsible, despite not possessing the license. 


DECISION 
Tidewater Homes Realty, Ince. protests the award of a contract to 


CitiWest under request for proposals (RFP) No. HO3R95062400000, a 
total small business set-aside, issued by the United States Department 
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of Housing and Urban Development (HUD), for real estate asset manager 
(REAM) services for single-family properties owned by HUD or in its 
custody in Virginia Beach, Virginia area under a firm, fixed-price, 
indefinite quantity, contract for a base year with 4 option years. 


We sustain the protest. 


The RFP, issued as a small business set-aside on June 1, 1995, 
provided for a best value award with the technical evaluation being 
worth more than cost/price. The RFP listed the following technical 
evaluation factors and corresponding values: 


1. Demonstrated experience in the management of single-family 
properties Similar to and in a like area as those covered by this 
Soetrcitatioln. {22 points] 


2. Demonstrated experience in developing lists of needed 
repairs, such as is reguired by HUD's Minimum Properly Standards 
(MPS), and estimating the cost of repairs. [25 points]. 


3. Demonstrated experience in soliciting repair bids, 
coordination and overseeing repair work, and inspecting for 
Sadtistacuory work completion. [15 points} 


4, Demonstrated experience in managing a rental program, 
including eslabiishing fair markel renlals and coileclions [rom 
present and former tenants, for single-family properties. [10 
points] 


5. Understanding of HUD objectives and the required tasks as 
specified in the solicitation. [10 points] 


6. Evidence of adequate office--staffed with appropriately 
trained staff and equipped appropriately (or the ability to 
establish such), reasonably located so as to provide convenient 
service to HUD and its clients in the area to be served, and to 
carry out all duties specified in the solicitation. [15 points] 


As part of their technical proposal, offerors were required to submit 
a completed Form 477, List of Repairs (included in RFP), for a 
specified property in Virginia Beach. Offerors were cautioned that: 


"Proposals submitted without this form will not be disqualified 
from competing for the award, but omission of the form may 
adversely affect the offeror's technicai points achieved in 
[factor 2)]." 


In addition the RFP required: 


"The contractor must supply with its Technical Proposal evidence 
of its Virginia Real Estate Broker License--in the contractor's 
name as it appears on the offer--to be determined responsible and 
eligible for award." [Emphasis in original. ] 


At the pre-proposal conference, a HUD official stated that if evidence 
of the foregoing license were not included in the proposal, "evidence 
of [the offeror's] ability to provide the license at award must be 





provided." The RFP expressly stated that proposals would be initially 
screened to ensure that they contained a Form 477 for the specified 
property and evidence of 4 Virginia Real Estate Broker License. 


HUD received 12 initial proposals hy November 13. Tidewater, whose 
principals were a manager and former employee of the local incumbent 
contractor performing these services, and CitiWest, from Tucson, 
Arizona, both submitted proposals. A technical evaluation panel 
(TEP), comprised of members from the regional contracting office in 
Philadelphia, Pennsylvania evaluated proposals. Five proposals, 
including those of Tidewater and CitiWest, were included in the 
competitive range. Three of the proposals, including CitiWest's, with 
a score of 91 points, were found technically acceptable and two, 
including Tidewater's, with a score of 37 points, were found capable 
of being made acceptable. 


By letter of June 21, 1996, HUD conducted discussions with the 
competitive range offerors, during which it advised Tidewater of the 
weaknesses and/or deficiencies in its proposal. HUD received best and 
final offers (BAFO) by July $3. CitiWest's BAFO at an evaluated unit 
price of $1,317 received the same technical score of 91 points. 
Tidewater's BAFO at an evaluated unit price of $1,233 received a 


technical score of 47 points. 


The TEP rated Tidewater's BAFO relatively low under five of the six 
evaluation factors, finding that Tidewater's proposal lacked pertinent 
detail in many respects. In contrast, CitiWest's proposal received 
high point scores under all of the factors and was determined to be 
reasonably priced. HUD determined that CitiWest's technically 
superior proposal was worth Lhe addilitonal cost because it would 
likely have fewer performance problems, resulting in less costs to the 
government, and made award to that firm on August 23. This protest 
followed. 


Tidewater protests that its proposal contained the required pertinent 
details and was misevaluated. Tidewater claims that the 
Philadelphia-based TEP was biased against Tidewater, as well as other 
locally based companies, and that HUD may have harbored resentment 
against Tidewater due to actions that Tidewater undertook in a prior 
procuremenl. Tidewater notes Lial tt submilled essenlially Lhe same 
proposals in response to HUD procurements for similar REAM services 
that a HUD Richmond-based TEP rated very favorably. Tidewater also 
claims that CitiWest did not have the Virginia Real Estate Broker 
License in its name at the time of award as required by the RFP. 


The evaluation of proposals is a matter within the discretion of the 
conlracling agency. Our Office will only question Lhe agency's 
evaluation where it lacks a reasonable basis or conflicts with the 
stated evaluation criteria for award. SC&A, Inec., B-270160.2, Apr. 

10, 1996, 96-1 CPD para. 197. The record must reasonably support the 
evaluation of the proposals, Intown Properties, Ine., B-262236.2; 
B-262237.2, Jan. 18, 1996, 96-1 CPD para. 89, and it is fundamental that 
the contracting agency must treat all offerors equally; it must 
even-handedly evaluate offers against common requirements and 
evaluation criteria. Sci-Tec Gauging, Inc.; Sarasota Measurements & 
Semerors, Ines, )6-252100, B-252406-2, guiness, 1°93, 93-1 CPD 494; 
SBeeure Servs. Technelegy, Inec., B-233059, Apr. 25, 1990, 90-1 CPD para. 
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421. As illustrated by the examples below, our review of the record 
reveals that CitiWest's and Tidewater's proposal were unegually 
evaluated under the RFP's evaluation factors. [1] 


For example, under Factor 2, Tidewater's proposal received 6 points, 
while CitiWest's proposal received 24 points. Tidewater's low score 
was attributed to an asserted lack of detail and clarity contained in 
the submitted Form 477 on the Virginia Beach property specified in the 
RFP and questions regarding Tidewater's ability to prepare cost 
estimates. These evaluated problems were brought to Tidewater's 
attention during discussions. In its BAFO, Tidewater basically 
explained, with some further elaboration, why it believed the 
submitted Fortin 477 and repair list satisfied the agency's requirements 
as stated in Factor 2[{2] and its method for preparing cost estimates. 
The TEP rated Tidewater's BAFO with the same score because the Form 
477 still did not contain sufficient detail and because Tidewater's 
initial response reyarding the preparation of cost estimates caused 
the TEP to question the validity and sincerity of Tidewater's response 
in its BAFO. 


Our review of Tidewater's BAFO indicates that the information 
contained in its Form 477 and its overall response to this factor 
contained much the same substantive detail as CitiWest's proposal. 
Based on our review of the two proposals, we cannot identify what 
specific details are missing from, or unclear in, Tidewater's Form 477 
with attachments. HUD does not explain what details are missing or 
what iS unclear; nor does it comment upon the accuracy of the needed 
repairs on the specified property as identified by Tidewater. 


On the other hand, while CitiWest's technical proposal seemed to 
address the same categories of information for repairing a specific 
property as required in a Form 477, no Form 477 was included in its 
proposal, despite the RFP's admonitions, and the property which was 
assessed by CitiWest under this factor was one that CitiWest was 
responsibie for in Camden, New Jersey (under a REAM contract 
apparently administered by the Philadelphia HUD office), rather than 
the Virginia Beach property identified in the RFP.[{3] We do not 
believe the evaluators could have reasonably rated CitiWest's proposal 
with close to the maximum score and much more favorably than 
Tidewater's proposal for this factor, given CitiWest's failure to 
provide a Form 477 for the specified property as was requested in the 
RFP. [4] 


Moreover, we find questionable HUD's explanation concerning its 
failure to credit Tidewater under Factor 2 for its apparently 
appropriate BAFO response as to who will prepare cost estimates for 
this work, since agencies are generally required to credit offerors 
for explanations in response to discussion questions. Intown 
Properties, Inc., supra. Under the circumstances, the record sugyests 
disparate evaluation of the two proposals under this factor. 


Another example of unequal treatment involves the evaluation of Factor 
6. The TEP awarded Tidewater's proposal a final score of 7 out of 15 
points for this factor and CitiWest's proposal 12 points. The ayency 
downgraded Tidewater's proposal because it was allegedly vague as to 
location of the office space and the division of responsibilities 
among staff members. Here again, our review indicales Lhal 
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Tidewater's BAFO and overall technical response to this factor was 
specific as to prospective locations, staff, and equipment. In 
contrast, CitiWest's proposal did not identify any specific location 
or staff in its proposal, but stated only how it would do so if 
awarded the contract. 


Finally, Tidewater argues, and our review confirms, that CitiWest did 
not provide a Virginia Real Estate Broker License with its proposal. 
While CitiWest proposed to promptly obtain the license after award, 
Tidewater notes that its investigation reveals that CitiWest has not 
obtained the license and HUD has not disputed this assertion. In any 
event, the requirement that the offeror furnish a specific license to 
be eligible for award was a definitive responsibility criterion that 
had to be satisfied as prereguisite for award.[5] RSI Realty Servs. 
Pnc., B-262238, Dec. 12, 1995, 95-2 CPD para. 252 (finding the same 
requirement imposed in a REAM service procurement conducted by the HUD 
Philadelphia office to be a definitive responsibility criterion which 
was waived for CitiWest). By finding CitiWest responsible and making 
award to that firm, the agency effectively waived the requirement for 
possession of the license prior Lo award.[6] Where an agency waives 
such a requirement, it is required to amend the RFP; an agency's 
failure to amend represents unequal treatment of the offerors. See 
iecley Realty Co., Inc., 65 Comp. Gen. 510 (1986), 86-1 CPD para. 398. 


In sum, as illustrated by the foregoing examples, we find that 
Tidewater's proposal was evaluated unequally vis-a-vis CitiWest's 
proposal. Thus, we cannot conclude that the award to CitiWest is 
reasonably supported, and we sustain the protest on this basis. 
Intown Properties, Inc., supra; Sci-Tec Gauging, Ine.; Sarasota 
Measurements & Controls, Inc., supra; Secure Servs. Tech., Inec., 
supra. 


We recommend that the agency determine whether the requirement that an 
offeror possess a Virginia Real Estate Broker License in order to 
receive award exceeded the agency's requirements; if it does, the 
agency should amend the RFP, obtain and evaluate new proposals, and 
make award in accordance with the revised RFP. If CitiWest is not the 
successful offeror, its contract should be terminated. Alternatively, 
if the license remains a requirement, we recommend that the agency 
reject CitiWest's proposal if CitiWest does not possess the requisite 
license (subject to Small Business Administration review under 
certificate of competency procedures, see FAR subpart 19.6) and 
terminate the contract, reevaluate the remaining competitive range 
proposals in accordance with the RFP, and make a new award selection. 
In any event, in light of the evaluation discrepancies noted, we 
recommend that a new TEP be appointed to evaluate the new proposals to 
assure equal evaluation. See J.M. Cashman, Inc., B-233773, Apr. 14, 
1989, 89-1 CPD para. 380. We further recommend that Tidewater be 
reimbursed the reasonable costs of filing and pursuing its protest 
under section 21.8(d)(1) of our Bid Protest Regulations, 61 Fed. Reg. 
39043 (1996) (to be codified at 4 C.F.R. sec. 21.8(d)(1)). The protester 
Should submit ils certified claim for costs to Lhe contracting agency 
within 60 days of receiving this decision pursuant to section 
21.8(f) (1) of our Regulations, 61 Fed. Reg. 39043 (to be codified at 4 
MePon. sec; Z2Il,ett) (1)). 


The protest is sustained. 
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Comptroller General 
of the United States 


1. Our discussion of the respective contents and evaluation of 
CitiWest's and Tidewater's proposals is necessarily general because we 
recommend a reevaluation of the proposals and because no protective 
order was issued, inasmuch as the protester did not employ legal 
counsel. 


2. Tidewater's BAFO noted the limited size on the one-page Form 477 
fer notations. 


3. The agency incorrectly states in its report in response to the 
protest that CitiWest's proposal contained a Form 477 "for the 
property specifically identified in the RFP." 


4, It appears that the Camden property was in need of much more 
extensive repairs than the specified Virginia Beach property, which 
permitted CitiWest to provide a more extensive description and perhaps 
receive more credit under this factor. 


5. A definitive responsibility criterion is a specific and objective 
standard established by an agency to measure an offeror's ability to 
perform the contract. Federal Acquisition Regulation (FAR) sec. 9-104-2. 
Such special standards put firms on notice that the class of 

prospective contractors is limited to those meeting qualitative or 
quantitative criteria deemed necessary for adequate performance, e.g., 
unusual expertise, specialized facilities, or particular licenses. 

fieson Mobilphone, Inc., B=258408.3, June 5, 1995, 95-1 CPD para. 267. 


6. The agency now states that the requirement that the license be 
obtained prior to award overstated its minimum needs and that the 
requirement as stated was "Sometimes confusing to offerors." The 
protester notes that this requirement may well have caused nonlocal 
firms not to compete and required other offerors to undergo the 
expense of obtaining the proper license prior to award. 





Decision 
Matter of: Lloyd H. Kessler, Inc. 
File: B-284693 
Date: May 24, 2000 
Jan D. Sokol, Esq., Stewart Sokol & Gray, for the protester. 
James L. Weiner, Esq., and Alton E. Woods, Esq., Department of the Interior, for the agency. 
Henry J. Gorczycki, Esq., and James Spangenberg, Esq., Office of the General Counsel, GAO, participated in the 


preparation of the decision. 


DIGEST 
Agency is required to disclose in the solicitation a subfactor to evaluate a particular type of experience under the 


experience factor where the subfactor constitutes 40 percent of the technical evaluation. 
DECISION 
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Lloyd H. Kessler, Inc. protests the award of a contract to John L. Jersey & Son, Inc. under request for proposals 
(RFP) No. FWS1-00-P004(KH), issued by the Department of the Interior, United States Fish and Wildlife Service 
(USFWS), for river dike renovation/rehabilitation at the Ridgefield National Wildlife Refuge, Washington. Kessler 
protests that the agency did not adhere to the stated RFP evaluation factors and improperly used unstated significant 
subfactors in the evaluation, and improperly evaluated Kessler’s past performance. 

We sustain the protest. 

The RFP, issued October 29, 1999, contemplated the award of a fixed-price construction contract. Contract 
performance cannot begin “until the eagle leaves its nest” in August 2000. RFP amend. 0002, at 1, 3. The RFP stated 
a best value evaluation plan under which “price and capability score are of equal importance.” RFP at 53. The 
capability score was said to be based on 100 possible points divided between (1) experience of firm (30 points) and 
(2) past performance of firm (70 points). RFP at 45. 

The RFP’s instructions for the preparation of responses to the past performance and experience factors stated that 
“OFFERORS SHOULD PROVIDE ONLY THE INFORMATION” requested by the RFP, that is, specified 
information about current contracts or contracts completed in the previous 2 years, including a brief description of 
the work performed and a point of contact. RFP at 46. The RFP stated that the agency would request performance 
information from the contract references to evaluate quality of performance, and may contact, and evaluate 
information rcccived from, rcfercnccs other than thosc identificd by offcrors. RFP at 47. Thc RFP spccificd five 
criteria that would be considered in evaluating past performance. RFP at 45-46. The RFP stated no criteria that 
would be considered in evaluating experience, but noted that this evaluation would be based on existing and prior 
contracts. RFP at 45-47. 

The agency received 17 proposals by the December 14 closing date. Kessler’s proposal offered the lowest price of 
$425,880 while Jersey’s proposed price was $487,221. Under the agency’s evaluation, Kessler’s proposal received a 
score of 70 points while Jersey’s proposal was one of the highest-rated proposals with a score of 90 points. Agency 
Report at 1; Tab 8, Evaluation Results. 

The agency’s actual evaluation differed from that stated in the RFP in that the relative weights for experience and 
past performance were reversed and experience was assigned 70 points and past performance was assigned 30 
points. Under this evaluation, the experience scores for Kessler and Jersey were 55 and 65 points, respectively, and 
the past performance scores were 15 and 25 points, respectively. ['] Agency Report at 1, Tab 8, Evaluation Results. 
The scores for experience were based on the following three criteria and relative weights not stated in the RFP: (1) 
bioengineered slope protection, 40 of the possible 70 points; (2) wetland excavation, 15 points; and (3) rip-rap 
construction, 15 points. Agency Report, Tab 7, Evaluation and Award Determination Summary, at 1. Kessler’s 
proposal received 25 of the possible 40 points for the bioengineered slope protection criterion, and received all of 
the combined 30 points available under the wetlands and rip-rap criteria. Supplemental Contracting Ofticer’s 
Statement (Apr. 10, 2000) at 5. The agency evaluation stated that Kessler “apparently has average experience in 
bioengineering (2.5 projects as the engineers can find in the experience information given) slope protection but has 
good wetland excavation and rip-rap placement experience.” Agency Report, Tab 7, Evaluation and Award 
Determination, at 2. 

Under past performance, Kessler’s proposal provided the requested information on 16 contracts completed in the 
past 2 years. Agency Report, Tab 3, Kessler’s Proposal, at 16-18. The agency based its past performance evaluation 
of Kessler on one of these contracts--a previous contract for dike repair with this agency--and two other contracts 
not identified in Kessler’s proposal. Agency Report, Tab 5, Performance Evaluations. The agency evaluation 
identified adverse comments concerning Kessler’s performance that resulted in a below-average score; however, the 
score was increased to an average score of 15 out of 30 points because the agency did not give Kessler the 
Opportunity to respond to the reports of adverse past performance. Agency Report, Tab 7, Evaluation and Award 
Determination, at 3. 

The agency’s cost/technical tradeoff analysis concluded that the concerns arising from reports of adverse past 
performance for Kessler were not worth the cost savings of Kessler’s lower price. Id. The agency determined that 
Jersey’s proposal, as the lowest priced of the most highly rated proposals, represented the best value. Id. at 4, 6. 

On February 4, 2000, the agency awarded the contract to Jersey. This protest followed. The agency suspended 
performance of Jersey’s contract pending resolution of the protest. 

Kessler protests the agency’s failure to adhere to the RFP’s stated evaluation weights and the use of unstated 
experience subfactors in the evaluation. 

It is fundamcnta! that offcrors must bc adviscd of the bascs upon which thcir proposals will bc cvaluated. H.J. Group 
Ventures, Inc., B-246139, Feb. 19, 1992, 92-1 CPD § 203 at 4. In particular, contracting agencies are required by 
statute and regulation to clearly set forth in the solicitation all evaluation factors and significant subfactors that will 
affect contract award and their relative importance. 41 U.S.C. § 253a(b)(1) (1994); Federal Acquisition Regulation § 
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15.304(d). An agency may not give importance to specific factors, subfactors, or criteria beyond that which would 
reasonably be expected by offerors. Kumasi Ltd./Kukawa Ltd. et al., B-247975.7 et_al., May 3, 1993, 93-1 CPD J 
352 at 6; Republic Realty Servs., Inc., B-242629, May 7, 1991, 91-1 CPD § 446 at 5. Contracting officials do not 
have the discretion to announce in the solicitation that they will use one evaluation plan and then follow another 
without informing offerors of the changed plan and providing them an opportunity to submit proposals on that basis. 
Kumasi Ltd./Kukawa Ltd. 

et_al., supra at 7. A solicitation that does not set forth a common basis for evaluating offers, which ensures that all 
firms are on notice of the factors for award and can compete on an equal basis, is materially deficient. The Faxon 
Co., B-227835.3, B-227835.5, Nov. 2, 1987, 87-2 CPD § 425 at 4. 

Here, contrary to the terms of the RFP, the agency considered experience to be the most important technical factor 
comprising over two-thirds of the capability score. The agency also states that the single subfactor, bioengineered 
slope protection, is the key consideration in the overall evaluation of experience, “as it will make or break this 
project,” Agency Report, Tab 7, Evaluation and Award Determination Summary, at 1 (italics in original). Consistent 
with this statement, bioengineered slope protection, though the term does not appear anywhere in the RFP, is the 
single-most important technical criterion, carrying more weight in the unstated evaluation plan--40 out of 100 
overall capability points--than past performance, which was stated to be the most important evaluation factor, or the 
weight assigned in the solicitation to the entire cxpericnec factor. This 1s a significant cvaluation subfactor that, by 
statute and regulation, must be clearly stated in the RFP along with its relative importance, even assuming it was 
considered reasonably related to the general experience factor stated in the RFP. See Kumasi Ltd./Kukawa Ltd., et 
al., supra, at 6 (subfactor four times more important than reasonably apparent from the RFP must be disclosed in 
RFP); Devres, Inc., B-224017, Dec. 8, 1986, 86-2 CPD J 652 at 3 (subfactor worth more than any other technical 
factor is “significant’); 

cf. Bulova Techs., LLC, B-281384, B-281384.2, Feb. 3, 1999, 99-1 CPD J 99, at 7-8 (agencies can properly take 
into consideration specific, albeit not expressly identified, experience in making qualitative distinctions between 
competing proposals, so long as the specific experience 1s logically encompassed by or related to the RFP’s 
requirements and stated basis for evaluation). 

The RFP did not otherwise indicate that “bioengineered slope protection” had the overwhelming significance given 
in the evaluation. As noted, this term is not mentioned anywhere in the RFP. The term appears in the Evaluation and 
Award Determination, which identifies this requirement as contract line item number (CLIN) 11 in the RFP, which 
is “Furnishing and Placing Fabric Slope Protection.” [*] Agency Report, Tab 7, Evaluation and Award Evaluation, at 
1; see RFP Bid Schedule at 1. Neither the estimated quantity for this CLIN and its associated price (as judged from 
the protester’s and awardee’s proposals), nor the description of this requirement in the statement of work indicates a 
ievel of significance approaching the weight accorded to bioengineered slope protection in the agency’s evaluation. 
RFP at C-108-16; Agency Report, Tabs 3 and 4, Proposals, Bid Schedule at 1. 

The agency conccdes that the actual rclative weights of the cvaluation factors were not consistcnt with that statcd in 
the RFP, but asserts that Kessler was not prejudiced by this evaluation because Kessler’s relative ranking would not 
improve if the weights announced in the RFP had been used. The agency’s arguments do not consider the fact the 
proposals were prepared without offerors knowing either the overwhelming significance of experience generally, or 
bioengineered slope protection specifically, in the evaluation, such that it cannot be said that offerors had a 
reasonable opportunity to compete on this basis. 

In this regard, Kessler prepared its proposal under the stated RFP evaluation plan whereby experience was the least 
important technical factor. As instructed by the RFP, Kessler provided only a brief description of the work 
performed under its prior contracts, and did not specifically identify any work involving bioengineered slope 
protection. Agency Report, Tab 3, Kessler’s Proposal, at 16-18. Even so, the agency was able to determine that 
Kessler “apparently has average experience” under this subfactor. Agency Report, Tab 7, Evaluation and Award 
Determination, at 2. Kessler states that, had it known when preparing its proposal that experience would be worth 70 
percent of the capability score, and that bioengineered slope protection was of prime importance in the evaluation, it 
would have placed greater emphasis on experience in the contents of its proposal, and specifically would have 
emphasized its experience providing bioengineered slope protection. Protester’s Comments at 9-10. Although the 
agency evaluation determined that Kessler performed bioengineered slope protection on “2.5 projects as the 
engineers can find in the experience information given”, Py Agency Report, Tab 7, Evaluation and Award 
Determination Summary, at 2, Kessler states that all of its projects involved this type of work and it would have 
provided this and other cxpcricncce information, had the agency stated that cxpcricncc and this subfactor were as 
significant as they were evaluated. Protester’s Supplemental Comments at 11. 

Given that Kessler has relevant experience, it is reasonable to conclude that Kessler’s score could increase 
considerably if the firm is permitted to submit a proposal with the knowledge of the actual importance of experience 
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generally, and of bioengineered slope protection specifically. [“] Because the agency used an evaluation plan not 
apparent from the RFP, the offerors, including Kessler, did not have the opportunity to compete on a common basis, 
so there is at least a reasonable possibility that Kessler’s lowest-priced proposal will be selected for award if the 
actual evaluation factors and subfactors are disclosed. [°] We therefore find that Kessler was prejudiced. 

We recommend that the agency amend the solicitation to state the agency’s proposal requirements and evaluation 
plan, request and evaluate revised proposals, and make a new source selection decision. (°] If a proposal other than 
Jersey’s is selected for award, the agency should terminate the contract previously awarded to that firm. We also 
recommend that the protester be reimbursed the reasonable cost of filing and pursuing its protest, including 
attorneys’ fees. 4 C.F.R. § 21.8(d)(1) (2000). The 

protester should submit its claim for costs, detailing and certifying the time expended and cost incurred, with the 
contracting agency within 60 days after receipt of this decision. 4 C.F.R. § 21.8(f)(1). 

The protest is sustained. 

Comptroller General 

of the United States 


Notes 

1. The agency also assigned risk ratings based on the offerors’ present and past performance. Kessler’s rating was 
high risk and Jersey’s rating was low risk. 

Agency Report, Tab 7, Evaluation and Award Determination, at 2-3. 

2. In its Supplemental Report, the agency references other RFP work items not included in CLIN 11 that constitute 
“bioengineering.” Supplemental Report at 2. However, the award document states that CLIN 11 was the “more 
weighted item” in the evaluation because it encompasses “bioengineered slope protection.” Agency Report, Tab 7, 
Evaluation and Award Determination, at 1. 

3. As stated, Kessler’s proposal did not identify bioengineered slope protection work. Neither the evaluation record 
nor the agency’s responses to this protest demonstrates how the agency made its determination in this regard as to 
Kessler’s proposal or any other proposal. We also question what .5 of a project means for purposes of this evaiuation 
and how that could be determined. No other offeror was noted to have experience on fractions of a project. 

4. It is possible that, under such circumstances, Kessler’s capability score could increase from 70 to 85 points from 
the evaluation of bioengineered slope protection alone, which would place its lowest-priced proposal extremely 
close to the highest-rated proposals, including Jersey’s, thus changing the underlying basis upon which the agency’s 
cost/technical tradeoff determination and source selection decision was made. 

5. We note that, even beyond the agency’s failure to disclose the actual evaluation plan in the RFP, the 
documentation supporting the evaluation is minimal, such that it would be difficult to determine the reasonableness 
of the evaluation, even if the RFP were not defective. The little documentation that does exist indicates that the 
agency unequaliy evaluated offerors under the bioengineered slope protection subfactor, in that, although the 
experience under the other subfactors for Kessler and another offeror were evaluated as good, that offeror had 
“minimal” biocnginccred slope protection cxpcricnce compared to Kessicr’s “average” cxpcricnce, but nevertheless 
received a considerably higher experience score than Kessler. Agency Report, Tab 7, Evaluation and Award 
Determination Summary. at 2, 5. 

6. Kessler also contests its past performance evaluation. While we do not decide whether the past performance 
evaluation was improper, the record reflects some problems that should be appropriately addressed when the agency 
reopens discussions. Specifically, the record reflects that the agency contacted only | of the 16 references listed in 
Kessler’s proposal (a USFWS contract), and while the agency alleges that the other references declined to comment 
on Kessler’s performance, there is no documentation supporting this statement, and Kessier has submitted evidence 
that no such contacts were made. The other two references considered by the agency were for contracts apparently 
completed more than 3 years before the evaluation, although the RFP requested information only on contracts 
performed in the last 2 years. In addition, Kessler has vigorously disputed the accuracy of the negative comments 
regarding its past performance, and the agency states that Kessler was not given the opportunity to respond to these 
negative comments. Since such matters should be raised, where, as here, discussions are to be conducted, the agency 
can now afford Kessler an opportunity to respond to any negative comments regarding its past performance. 
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